0

I'm in the process of upgrading Apache Solr from version 8.6.0 to 9.8.0, and part of this involves upgrading the Lucene index format.

I have a large index directory (~650 GB), and I'm running the IndexUpgrader tool provided by Lucene to upgrade the existing index data.

Here’s the command I used:

java -Xmx8G -cp lucene-core-8.11.4.jar:lucene-backward-codecs-8.11.4.jar \
org.apache.lucene.index.IndexUpgrader \
-delete-prior-commits -verbose /mnt/solr/gcp_data/data/index

After running for a while, the process fails with the following error:

    delGen=1 :id=4l4qwonvoyjybn7ogcbfptr8u _4mykq(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750433990252}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptqji _4myqe(8.6.0):c28/1:[diagnostics={os=Linux, java.vendor=Red Hat, Inc., java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=merge, mergeFactor=10, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446288277}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1 :id=4l4qwonvoyjybn7ogcbfptr74 _4myq0(8.6.0):c27/1:[diagnostics={os=Linux, java.vendor=Red Hat, Inc., java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=merge, mergeFactor=10, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446169024}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1 :id=4l4qwonvoyjybn7ogcbfptr5i _4myn2(8.6.0):c46/1:[diagnostics={os=Linux, java.vendor=Red Hat, Inc., java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=merge, mergeFactor=10, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750437328313}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1 :id=4l4qwonvoyjybn7ogcbfptqt8 _4myqx(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446442343}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr97 _4myqy(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446442690}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr9b _4myqz(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446463537}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr9f _4myqo(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446362851}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr86 _4myqw(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446424317}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr93 _4myr3(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446492533}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr9w _4myqh(8.6.0):C1:[diagnostics={java.vendor=Red Hat, Inc., os=Linux, java.version=1.8.0_432, java.vm.version=25.432-b06, lucene.version=8.6.0, os.arch=amd64, java.runtime.version=1.8.0_432-b06, source=flush, os.version=4.18.0-553.51.1.el8_10.x86_64, timestamp=1750446303287}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] :id=4l4qwonvoyjybn7ogcbfptr7d into _4myr9 [maxNumSegments=1]
            at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2136)
            at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:2069)
            at org.apache.lucene.index.IndexUpgrader.upgrade(IndexUpgrader.java:172)
            at org.apache.lucene.index.IndexUpgrader.main(IndexUpgrader.java:78)
    Caused by: java.lang.OutOfMemoryError: Java heap space
            at java.base/java.lang.StringUTF16.newBytesFor(StringUTF16.java:53)
            at java.base/java.lang.String.<init>(String.java:577)
            at java.base/java.lang.String.<init>(String.java:1425)
            at org.apache.lucene.codecs.StoredFieldsWriter$MergeVisitor.stringField(StoredFieldsWriter.java:187)
            at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:285)
            at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:690)
            at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.merge(CompressingStoredFieldsWriter.java:642)
            at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:228)
            at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
            at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4757)
            at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4361)
            at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5920)
            at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
            at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)

I am using 8 core and 32GB Linux Rocky9 machine.

What I’ve already tried:

Ensured the data directory is not live (copied from original and not served by Solr during upgrade).

System has 32 GB RAM and 20 GB swap configured.

Tried increasing heap to 8G and later 16G and 20G (-Xmx20G) but still encountering OOM in some merge phases.

Observed via dmesg that the kernel OOM killer has sometimes terminated the Java process.

My Questions:

Is there a recommended memory configuration for IndexUpgrader with large index sizes like ~650GB?

Can the merge process be parallelized or optimized to avoid memory exhaustion?

Are there any Lucene/Solr-specific flags or best practices to safely handle large index upgrades?

Any insight into handling this kind of scenario is appreciated.

Improve this question
Related questions
597
How to deal with "java.lang.OutOfMemoryError: Java heap space" error?
4
DateTimeParseException: fails on one host, works on another, same JDK
150
java.lang.OutOfMemoryError: Java heap space in Maven
Load 3 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.