Im able to create the CSV using the solution provided here:
Export Elasticsearch results into a CSV file
but problem arises when the records exceeds 10000 (size=10000), is there any way to write all the records?
2
-
I would suggest to look into pagination, or the scroll api. The elasticsearch-py scroll helper might be useful here. Then you can post details about where it's going wrong for you, and people might be able to help.dshockley– dshockley2017-11-09 07:59:12 +00:00Commented Nov 9, 2017 at 7:59
-
thanks, looking into the scroll api , that might solve my problem.sand33p_23– sand33p_232017-11-09 08:59:03 +00:00Commented Nov 9, 2017 at 8:59
Add a comment
|
1 Answer
The method you given in your question use elasticsearch's Python API, and es.search do have a 10 thousand docs retrieving limit.
If you want to retrieve data more than 10,000, as suggested by dshockley in the comment, you can try scroll API. Or you can try elasticsearch's scan helpers, which automates a lot work with scroll API. For example, you won't need to get a scroll_id and pass it to the API, which will be necessary if you use scroll directly.
When use helpers.scan, you need to specify index and doc_type in the parameters when call the function, or write them in the query body. Note that, the parameter name is 'query' rather than 'body'.
1 Comment
sand33p_23
thanks, using scroll api im able to get all the results in the csv file.
