Apache Spark: loading multiple files or a directory in java

Question

I am trying to load multiple files/directories in SPARK using Java, I have found a few examples on how to do this in scala, can someone give an example with explanation on how to do this in Java?

In particular I would like to use regex like paths, so that I do not have to specify a fully qualified name for each file. I can already give a comma separated file values with fully qualified names.

I am loading from the local file system, I don't know if this makes a difference

The following is the code I have used to load the files:

    SparkConf sparkConf = new SparkConf().setAppName("TableAggregator");
    JavaSparkContext ctx = new JavaSparkContext(sparkConf);
    JavaRDD<String> lines = ctx.textFile(args[0], 1);

Community · Accepted Answer · 2020-06-20 09:12:55Z

3

In Spark, the method textFile() takes an URI for the file (either a local path on the machine or a hdfs://, etc URI).

You can run this method on directories, compressed files and wildcard :

ctx.textFile("data.txt");

ctx.textFile("/your/directory/");

ctx.textFile("/your/directory/*");

ctx.textFile("/your/directory/*.gz");

Be aware that when you use a path for your input, it has to be the same path for all the worker nodes. So you have to copy the file to all workers or use a shared network-mounted file system.

So you can use a pattern with the wildcard to do it simply.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Dec 12, 2014 at 9:14

ahars

2,7961 gold badge12 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

tsar2512 Over a year ago

Thanks for your response.. is the directory look up recursive?

ahars Over a year ago

For me, it is not recursive. You have a exception java.io.IOException: not a file

Kristian Over a year ago

@AntoineHars Where is this documented?

ahars Over a year ago

Unfortunately, I haven't found it on the documentation but I tried by myself and I get this error. If you search the term "textfile" in the Spark programming Guide (spark.apache.org/docs/latest/programming-guide.html) and in the javadoc (spark.apache.org/docs/latest/api/java/index.html), there is no mention of the behavior in case of nested directories

Collectives™ on Stack Overflow

Apache Spark: loading multiple files or a directory in java

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related