How can I use scan/scroll with pagination and sort in ElasticSearch?

Question

I have a ES DB storing history records from a process I run every day. Because I want to show only 20 records per page in the history (order by date), I was using pagination (size + from_) combined scroll, which worked just fine. But when I wanted to used sort in the query it didn't work. So I found that scroll with sort don't work. Looking for another alternative I tried the ES helper scan which works fine for scrolling and sorting the results, but with this solution pagination doesn't seem to work, which I don't understand why since the API says that scan sends all the parameters to the underlying search function. So my question is if there is any method to combine the three options.

Thanks,

Ruben

scribu · Accepted Answer · 2020-03-04 11:21:43Z

5

When using the elasticsearch.helpers.scan function, you need to pass preserve_order=True to enable sorting.

(Tested using elasticsearch==7.5.1)

answered Mar 4, 2020 at 11:21

scribu

3,0984 gold badges35 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

bfontaine Dec 27, 2024 at 13:28

The documentation for this argument says that "this can be an extremely expensive operation and can easily lead to unpredictable results".

Carlos Rodriguez · Accepted Answer · 2016-01-07 15:11:13Z

1

yes, you can combine scroll with sort, but, when you can sort string, you will need change the mapping for it works fine, Documentation Here

In order to sort on a string field, that field should contain one term only: the whole not_analyzed string. But of course we still need the field to be analyzed in order to be able to query it as full text.

The naive approach to indexing the same string in two ways would be to include two separate fields in the document: one that is analyzed for searching, and one that is not_analyzed for sorting.

"tweet": { 
    "type":     "string",
    "analyzer": "english",
    "fields": {
        "raw": { 
            "type":  "string",
            "index": "not_analyzed"
        }
    }
}

The main tweet field is just the same as before: an analyzed full-text field.

The new tweet.raw subfield is not_analyzed.

Now, or at least as soon as we have reindexed our data, we can use the tweet field for search and the tweet.raw field for sorting:

GET /_search
    {
        "query": {
            "match": {
                "tweet": "elasticsearch"
            }
        },
        "sort": "tweet.raw"
    }

answered Jan 7, 2016 at 15:11

Carlos Rodriguez

8839 silver badges13 bronze badges

6 Comments

ruba Over a year ago

No I can't. If I use the scroll command the results I get are not sorted. Also I am not sorting on a string field.

Carlos Rodriguez Over a year ago

You tried change the field mapping to not_analyzed? What field type you will sort?

ruba Over a year ago

Yes I tried changing it to not_analyzed but still doesn't work. I am filtering by date

Carlos Rodriguez Over a year ago

When you see the mappping? It is {"type":"date","format":"dateOptionalTime"}?

ruba Over a year ago

Yes it is. But I don't think the problem is with the format of the field I am sorting with but the combination of sort, scroll and pagination. I can sort if I don't use scroll and I can use pagination with search but not when ordering...

|

Collectives™ on Stack Overflow

How can I use scan/scroll with pagination and sort in ElasticSearch?

2 Answers 2

1 Comment

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related