Spark dataframe will not show() - Py4JJavaError: An error occurred while calling o426.showString

Question

I have a dataframe that I cannot .show(). Every time it gives the following error? Is it possible that there is a corrupted column?

Error:

Py4JJavaError: An error occurred while calling o426.showString. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 381.0 failed 4 times, most recent failure: Lost task 0.3 in stage 381.0 (TID 19204, ddlps28.rsc.dwo.com, executor 99): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2/python/pyspark/worker.py", line 177, in main

@AliAzG - I am using 2.2.0.cloudera1 . This dataframe will .show() prior to running a UDF (via Spark SQL) on the dataframe, but there are no errors. — user1624577
– user1624577, Commented Dec 6, 2018 at 20:04

benlaird · Accepted Answer · 2018-12-06 20:54:20Z

3

Your error most likely isn't actually in the "show" operation. It's that .show is what triggers execution of your DAG. You said it works if you don't run your UDF, you probably just have a different error in that UDF. The log would probably be on the worker nodes, so try access through your Hadoop UI to get access to executor logs to see what really is breaking

answered Dec 6, 2018 at 20:54

benlaird

8797 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark dataframe will not show() - Py4JJavaError: An error occurred while calling o426.showString

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related