Spark OutOfMemoryError when taking a big input file

Question

I have a spark application that reads a file with 100 million lines (each line has a code, such as US1.234.567B1) and gets some patterns out of it, as follows:

  val codes = sc.textFile("/data/codes.txt")

  def getPattern(code: String) = code.replaceAll("\\d", "d")

  val patterns: RDD[(String, Int)] = codes
    .groupBy(getPattern)
    .mapValues(_.size)
    .sortBy(- _._2)

  patterns
    .map { case (pattern, size) => s"$size\t$pattern" }
    .saveAsTextFile("/tmp/patterns")

I am running this on master=local[*], and it fails with java.lang.OutOfMemoryError: GC overhead limit exceeded.

Why is that?

I thought that Spark can handle any size of input, as long as it has enough hard disk space.

two docs explaining why groupBy should be avoided: databricks.gitbooks.io/databricks-spark-knowledge-base/content/… github.com/awesome-spark/spark-gotchas — David Portabella
– David Portabella, Commented Oct 1, 2016 at 7:46

zero323 · Accepted Answer · 2016-09-30 18:17:54Z

2

Long short you're trying to use Spark anti-pattern:

.groupBy(getPattern)
.mapValues(_.size)

that can be easily expressed for example as:

codes.keyBy(getPattern).mapValues(_ => 1L).reduceByKey(_ + _).sortBy(_._2, false)

I thought that Spark can handle any size of input.

It usually can scale out as long as you don't make it impossible. group / groupByKey on RDDs create local collections for each key. Each of these hast to in the memory of a single executor.

answered Sep 30, 2016 at 18:17

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ganeiy · Accepted Answer · 2016-09-30 18:20:12Z

1

Yes spark can process very large files, but the unit of parallelism is the executor. 'Out of memory error' is because the spark executor memory or the spark driver memory is insufficient. Please try increasing spark.executor.memory and spark.driver.memory and also tune the number of executors before you submit the job.

You can set these values in a property file or in SparkConf or directly in command line during spark-submit. Link http://spark.apache.org/docs/latest/configuration.html

answered Sep 30, 2016 at 18:20

ganeiy

3022 silver badges9 bronze badges

Collectives™ on Stack Overflow

Spark OutOfMemoryError when taking a big input file

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related