java.lang.OutOfMemory:Java heap space error in spark-submit

Question

I am running Spark application using spark-submit and defined JVM parameters. With this set of parameters I get java heap space error:

EXTRA_JVM_FLAGS="-server -XX:+UseG1GC
                         -XX:ReservedCodeCacheSize=384m
                         -XX:MaxDirectMemorySize=2G
                         -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
     --master "local[4]"
     --driver-memory 2G
     --driver-java-options "${EXTRA_JVM_FLAGS}"

I tried to increase driver memory but it caused JVM crash. Also, I tried to increase max direct memory size which did not help in any way. What options should I change to fix heap space error?

Juraj Martinka · Accepted Answer · 2019-09-23 06:00:28Z

1

You should try the most basic option -Xmx - this is the max heap space size. Code cache and direct memory size are native memory areas and don't affect the size of the heap. By default, the JVM allocates 1/4 of RAM available on the box as max heap size. You can increase that if the machine is dedicated to the one JVM process pretty safely.

answered Sep 23, 2019 at 6:00

Juraj Martinka

4,4152 gold badges27 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Cassie Over a year ago

As far as I know, it works a bit different for Spark. Specifically, -Xmx is defined through the driver-memory option

Collectives™ on Stack Overflow

java.lang.OutOfMemory:Java heap space error in spark-submit

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related