3

I have 4 machines, M1, M2, M3, and M4. The scheduler, client, worker runs on M1. I've put a csv file in M1. Rest of the machines are workers.

When I run the program with read_csv file in dask. It gives me Error, file not found

Improve this question

1 Answer 1

Reset to default
3

When one of your workers tries to load the CSV, it will not be able to find it, because it is not present on that local disc. This should not be a surprise. You can get around this in a number of ways:

  • copy the file to every worker; this is obviously wasteful in terms of disc space, but the easiest to achieve
  • place the file on a networked filesystem (NFS mount, gluster, HDFS, etc.)
  • place the file on an external storage system such as amazon S3 and refer to that location
  • load the data in your local process and distribute it with scatter; in this case presumably the data was small enough to fit in memory and probably dask would not be doing much for you.
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

So will the similar problem arise in to_csv. I mean the workers will write their portion of computed file in their machines?
It is possible in dask, that worker nodes fetch some (or whole file) itself from the client, when needed?
Yes, you can upload files from the client
Will Dask do it automatically or I have to do some configuration?
How would dask know to do that? Also, note that this is not the intended use of file_upload, you would be better off sorting out your own copies, if copying is the method you want to use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.