Scala getting java exception while trying to create RDD which has more than 255 columns

Question

I have a huge data set with almost 600 columns but, while I am trying to create a DF it is failing with

Exception in thread "main" java.lang.ClassFormatError: Too many arguments in method signature in class file

Sample code:

def main(args: Array[String]): Unit = {
  val data = sc.textFile(file);
  val rd = data.map(line => line.split(",")).map(row => new Parent(row(0), row(1), ........row(600)))
  rd.toDF.write.mode("append").format("orc").insertInto("Table")
}

Can someone help how to perform workaround for this?.

Did you try reading the data directly as a DF with spark.read.csv()? — Shaido
– Shaido, Commented Jun 1, 2018 at 5:50
Java does not support methods with more than 55 for static and 254 for non-static methods. You can change your Parent to have a List[String] as the single member. — sarveshseri
– sarveshseri, Commented Jun 1, 2018 at 6:39
I dont think I can use CSV here, some other cases I might go with different delimiter. — sande
– sande, Commented Jun 1, 2018 at 16:44

Ra41P · Accepted Answer · 2018-06-02 17:24:12Z

2

I believe there is a limit on maximum method arguments for a Java object, which therefore extends to Scala object as well. A Person class with 600 params would be unfeasible.

The most ideal solution would be to read the csv natively as:

spark.read.csv(filePath)

Additionally, you may choose to increase the maxColumns option, using the signature.

spark.read.options().csv()

While it does not directly affect your use-case, the max-columns is set to 20480. More information on these parameters can be found here.

edited Jun 2, 2018 at 17:24

answered Jun 1, 2018 at 7:45

Ra41P

7741 gold badge9 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sande Over a year ago

What if my delimiter is not comma instead of that i have some delimiter like "~"

Ra41P Over a year ago

You can use the "sep" to change the separator to whichever character you desire while using spark CSV. Look at the java/scaladoc linked in the answer for more information on the sep option.

Collectives™ on Stack Overflow

Scala getting java exception while trying to create RDD which has more than 255 columns

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related