0

I am not using PySpark . I am using scala to run my spark application. I execute a python script within the spark application by calling calling RDD.pipe. I want to be able to log something from the python script ,and see the logged information in the executor log. I am not using PySpark. I am executing a Python script within a Spark application via the API RDD.pipe.

I want to somehow provide the user the ability to debug the script by printing some messages at the level of the Python script and have those Python scripts available at the level of the Spark logger.

How can I do that ?

Improve this question
2
Related questions
71
How do I log from my Python Spark script
4
Spark Pipe example
74
Apache Spark logging within Scala
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.