0

I am trying to run a Python Flink Application on the standalone Flink cluster. The application works fine on a single node cluster but it throws the following error on a multi-node cluster. java.lang.Exception: The user defined 'open()' method caused an exception: An error occurred while copying the file. Please help me resolve this problem. Thank you

The application I am trying to execute has the following code.

from flink.plan.Environment import get_environment
from flink.plan.Constants import INT, STRING, WriteMode

env = get_environment()

data = env.from_elements("Hello")

data.map(lambda x: list(x)).output()
env.execute()
Improve this question
2
  • Please provide code for your open() method and the full stacktrace. With information you provided so far it is not possible to help you. Commented Feb 27, 2018 at 9:49
  • The open() function is actually not user defined but I don't seem to find a solution for this problem. This problem persists only when executed in a multi-node cluster. Commented Feb 27, 2018 at 10:02

1 Answer 1

Reset to default
1

You have to configure "python.dc.tmp.dir" in "flink-conf.yaml" to point to a distributed filesystem (like HDFS). This directory is used to distributed the python scripts.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Hi Chesnay, thanks for the reply. Is there way I can set it up without a HDFS system?
at the moment you unfortunately require some kind of distributed filesystem.
I have a NFS file system and the error still persists even after configuring "python.dc.tmp.dir" in the flink-conf.yaml to the shared folder. Please help. Thank you

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.