"java.lang.IllegalAccessError" while reading multiline json file in Spark2.0

Question

while reading multiline json file in Spark2.0 getting exception

val data = spark.read
    .option("multiline",true)
    .json("C:\\user\\Spark\\DataSets\\employees_multiLine.json")

Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.()V from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:262) at org.apache.spark.input.StreamFileInputFormat.setMinPartitions(PortableDataStream.scala:51) at org.apache.spark.rdd.BinaryFileRDD.getPartitions(BinaryFileRDD.scala:51) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)

can you pls share your sample json data? I can attempt to build the solution using actual data. — Shantanu Kher
– Shantanu Kher, Commented Jul 19, 2020 at 12:47
From javadoc: this error can only occur at run time if the definition of a class has incompatibly changed. If you got it at run time, maybe you has invalid library dependence tree. There is any IDE or SBT warning? — Mikhail Ionkin
– Mikhail Ionkin, Commented Jul 19, 2020 at 15:50
@MikhailIonkin -- By adding google guava dependency it got resolved <dependency> <groupId>com.google.guava</groupId> <artifactId>guava</artifactId> <version>15.0</version> </dependency> — Nisha Gupta
– Nisha Gupta, Commented Sep 27, 2020 at 5:24
@ShantanuKher here you go with nested json [ { "name": "Nish", "product": "Headphone", "Delivery": { "name": "Nisha", "address": "Chennai", "mob": "1234567" } }, { "name": "Gupta", "product": "T-shirt", "Delivery": { "name": "Guptans", "address": "Bangalore", "mob": "5738612" } } ] — Nisha Gupta
– Nisha Gupta, Commented Sep 27, 2020 at 5:31

Manjula N · Accepted Answer · 2020-11-03 13:11:03Z

2

Updating hadoop to 2.7.2 or higher will resolve this.

This issue is explained here in detail https://stackoverflow.com/a/36443787

answered Nov 3, 2020 at 13:11

Manjula N

462 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

"java.lang.IllegalAccessError" while reading multiline json file in Spark2.0

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related