Count documents per day using NEST and ElasticSearch

Question

I'm trying to implement an ElasticSearch query using NEST, to be able to produce a result of key/value pairs where key is a date and value is a count. The documents I'm querying all have a created_date and I want to count how many documents where inserted per day, during the last 7 days or similar.

I've checked out the Count method of IElasticClient, but that seem to give me a total count and not per day. I'm thinking I need to do a facet on the data, but can't quite figure out how to implement it.

Any help would be appreciated :)

Martijn Laarman · Accepted Answer · 2013-10-01 18:15:23Z

1

Going with facets is indeed the way to go:

http://www.elasticsearch.org/guide/reference/api/search/facets/date-histogram-facet/

public class Doc
{
    public string Id { get; set; }
    public DateTime CreatedOn { get; set; }
}

public void TempFacetExample()
{
    var result = this._client.Search<Doc>(s => s
        .FacetDateHistogram(fd => fd
            .OnField(p => p.CreatedOn)
            .Interval(DateInterval.Day)
            //global forces it to count out of scope
            //from the main query (if any)
            .Global()
            .FacetFilter(ff => ff
                .Range(rf => rf
                    .From(DateTime.UtcNow.AddDays(-7))
                    .To(DateTime.UtcNow)
                )
            )
        )
    );
    var facetBucket = result.Facet<DateHistogramFacet>(p => p.CreatedOn);
    //facetBucket.Items now holds the days with counts
    //if I remember correctly elasticsearch wont return empty buckets
    //so you have to handle missing days (that have no docs).

}

answered Oct 1, 2013 at 18:15

Martijn Laarman

13.5k47 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alex Brasetvik Over a year ago

Note: That filter is not cachable. Depending on the indexing flow and search patterns, it might be a good idea to partition into day-specific indexes and only search over the indexes for the relevant days.

Alex Brasetvik · Accepted Answer · 2013-10-01 18:10:13Z

What you want is a date_histogram-facet.

Here's an example that should cover the various ways I can interpret what you want:

export ELASTICSEARCH_ENDPOINT="http://localhost:9200"

# Index documents
curl -XPOST "$ELASTICSEARCH_ENDPOINT/_bulk?refresh=true" -d '
{"index":{"_index":"play","_type":"type","_id":1}}
{"created_date":"2013-09-30T12:00:00Z","key":"foo","count":12}
{"index":{"_index":"play","_type":"type","_id":2}}
{"created_date":"2013-09-30T13:00:00Z","key":"bar","count":14}
{"index":{"_index":"play","_type":"type","_id":3}}
{"created_date":"2013-10-01T12:00:00Z","key":"foo","count":42}
{"index":{"_index":"play","_type":"type","_id":4}}
{"created_date":"2013-10-01T14:00:00Z","key":"foo","count":13}
'


# Do searches

curl -XPOST "$ELASTICSEARCH_ENDPOINT/_search?pretty" -d '
{
    "facets": {
        "foos_per_interval": {
            "date_histogram": {
                "key_field": "created_date",
                "value_field": "count",
                "interval": "day"
            },
            "facet_filter": {
                "term": {
                    "key": "foo"
                }
            }
        }
    }
}
'

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 4,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "play",
      "_type" : "type",
      "_id" : "1",
      "_score" : 1.0, "_source" : {"created_date":"2013-09-30T12:00:00Z","key":"foo","count":12}
    }, {
      "_index" : "play",
      "_type" : "type",
      "_id" : "2",
      "_score" : 1.0, "_source" : {"created_date":"2013-09-30T13:00:00Z","key":"bar","count":14}
    }, {
      "_index" : "play",
      "_type" : "type",
      "_id" : "3",
      "_score" : 1.0, "_source" : {"created_date":"2013-10-01T12:00:00Z","key":"foo","count":42}
    }, {
      "_index" : "play",
      "_type" : "type",
      "_id" : "4",
      "_score" : 1.0, "_source" : {"created_date":"2013-10-01T14:00:00Z","key":"foo","count":13}
    } ]
  },
  "facets" : {
    "foos_per_interval" : {
      "_type" : "date_histogram",
      "entries" : [ {
        "time" : 1380499200000,
        "count" : 1,
        "min" : 12.0,
        "max" : 12.0,
        "total" : 12.0,
        "total_count" : 1,
        "mean" : 12.0
      }, {
        "time" : 1380585600000,
        "count" : 2,
        "min" : 13.0,
        "max" : 42.0,
        "total" : 55.0,
        "total_count" : 2,
        "mean" : 27.5
      } ]
    }
  }
}

I awarded Martijn the answer cause the question were about NEST. But your answer were just the thing I was looking for. Thank you so much!

Collectives™ on Stack Overflow

Count documents per day using NEST and ElasticSearch

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related