2

I have been using PySpark and have a problem with the logging. Logs from the Spark module are piped to STDOUT and I have no control over that from Python.

For example, logs such as this one are being piped to STDOUT instead of STDERR:

2018-03-12 09:50:10 WARN Utils:66 - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.

Spark is not installed in the environment, only Python and Pyspark.

How do I:

A. Redirect all logs to STDERR

OR

B. If that is not possible, disable the logs.

Things I have tried:

  1. I have tried to use the pyspark.SparkConf() but nothing I configure there seems to work.
  2. I have tried creating SparkEnv.conf and setting the SPARK_CONF_DIR to match just to check if I could at least disable the example log above, to no avail.
  3. I have tried looking at the documentation but no indication of how to accomplish what I am trying.
Improve this question
2
  • Or C. Redirect everything to /dev/null? Commented Mar 12, 2018 at 10:16
  • No, my program has actual STDOUT information that I want to display, these logs should not be going to STDOUT. Commented Mar 12, 2018 at 10:20

1 Answer 1

Reset to default
1

You can set Log Level to ERROR, so it will only show ERROR logs:

sc.setLogLevel("ERROR")  # sc is a SparkContext() object from the pyspark lib

But if you want to disable all PySpark logs you can do this:

sc.setLogLevel("OFF")

Check this Stack Thread

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

I am sorry, but where is sc object defined? what is it?
The sc object is the SparkContext. If you are using PySpark integrated with some IDE, automatically it starts a new spark context and provides you a SparkContext object named sc, and a SparkSession named spark. If you are not using PySpark integrated, and maybe importing libraries by hand, the sc object I refer is the object that you create with SparkContext() or SparkContext.getOrCreate() or similar instructions.
Thanks, and yes, I have been doing it manually. It seems to solve my problem. Thanks a bunch.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.