Query Failed [Failed to execute main query] OutOfMemoryError: Java heap space

Question

Hi i have data in elastic 80 M of data have 52 Go and i search in text with this query

$json_doc = '{
                 "query": {
                    "query_string": {
                        "query": "*' . $sujet . '*",
                        "lenient": true
                    }
                    },
     "size": 10000000,
     "from": 0,
     "fields": [
     "association_key","text"
                    ]
                }';

when I search a word that is in the base is looking less 20K I have my data but as his exceeds 20 K I have this error

[2015-11-14 20:03:04,663][DEBUG][action.search.type       ] [Pixx] [myindex3][2], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][2]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:56)
    at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:64)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:275)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:135)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:133)
    at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:262)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:490)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more
[2015-11-14 20:03:11,764][DEBUG][action.search.type       ] [Pixx] [myindex3][0], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][0]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:187)
    at org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:81)
    at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
    at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more

how i can resolve this problem i have dedicate server contains just a Intel Xeon W3520 c 4/8 t + 2.66 GHz RAM 32 GB ECC under Debian

Alain O'Dea · Accepted Answer · 2020-02-15 14:10:00Z

2

UPDATE: ES_HEAP_SIZE is no longer a documented option

Elastic Search won't use all of that memory without being configured.

You need to configure and tune Elastic Search and Debian's memory settings.

You'll need to configure one of these:

/etc/elasticsearch/jvm.options: a file containing JVM options (file is at config/jvm.options (in a TAR or ZIP distro rather than a Debian package)
ES_JAVA_OPTS: an environment variable with additional JVM settings including memory and GC tuning if needed (pretty much what's in jvm.options but in a single line separated by spaces)

Here's the relevant excerpt from Elastic's latest documentation as of February 2020:

Here are examples of how to set the heap size via the jvm.options file:
-Xms2g ①
-Xmx2g ②
① Set the minimum heap size to 2g.

② Set the maximum heap size to 2g.

It is also possible to set the heap size via an environment variable. This can be done by commenting out the Xms and Xmx settings in the jvm.options file and setting these values via ES_JAVA_OPTS:
ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch ①
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch ②
① Set the minimum and maximum heap size to 2 GB.

② Set the minimum and maximum heap size to 4000 MB.

See more here:

edited Feb 15, 2020 at 14:10

answered Nov 14, 2015 at 19:53

Alain O'Dea

21.8k1 gold badge64 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

eebbesen Over a year ago

Running against a Dockerized Elasticsearch 2.3.2 I found that setting ES_HEAP_SIZE alone didn't change my instance's heap size from the default. Adding ES_JAVA_OPTS did indeed change it, and seemed to ignore a lower setting for ES_HEAP_SIZE.

Alain O'Dea Over a year ago

Good catch @eebbesen! I've updated as ES_HEAP_SIZE is no longer a documented option. Thank you!

Community · Accepted Answer · 2017-05-23 12:03:57Z

1

Seems like you don't have enough space for Java's heap. Since it seems like you'd have more hardware wise you may try to increase it: Increase heap size in Java

edited May 23, 2017 at 12:03

CommunityBot

11 silver badge

answered Nov 14, 2015 at 19:32

su-ex

1067 bronze badges

Collectives™ on Stack Overflow

Query Failed [Failed to execute main query] OutOfMemoryError: Java heap space

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related