0

Getting an exception when trying to rename a file within Spark application. Permission denied - new file name. The same thing works good with the spark-shell with by the same user. P.S. The path is mounted to S3.

The code:

import org.spark_project.guava.io.Files
Files.move(new File(oldfilename), new File(newfilename))

The exception:

java.io.FileNotFoundException: /path/new_file_name.csv (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at org.spark_project.guava.io.Files$FileByteSink.openStream(Files.java:223)
    at org.spark_project.guava.io.Files$FileByteSink.openStream(Files.java:211)
    at org.spark_project.guava.io.ByteSource.copyTo(ByteSource.java:203)
    at org.spark_project.guava.io.Files.copy(Files.java:436)
    at org.spark_project.guava.io.Files.move(Files.java:651)

Expectations: file renamed

Improve this question
3
  • 1
    Rather obviously - permission denied. Error messages tend to mean what they say they mean. Especially if you provide no detail beyond 'this is the error I got'. You may get more detailed information (but I doubt it) if you stop using obsolete APIs; java.io.File is obsolete (and one of the reasons it is obsolete, is bad error messages, so relevant here). Use java.nio.file.Path objects and java.nio.file.Files.move (no need for guava) instead to possibly get a more detailed error message. Commented Apr 13, 2023 at 13:51
  • 1
    Some obvious things to check: The directory you are writing to has to exist, Files.move (neither guava's nor java.nio's) will make them for you. And, of course, check the permissions on that folder. The error seems to occur on the creation part, not on the 'remove the old one' part. Commented Apr 13, 2023 at 13:51
  • 2
    Update. The issue was resolved by our DevOps engineer. There was something wrong with the fuse config on s3fs. Commented Apr 19, 2023 at 17:44

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.