How can I read multiple files from hdfs using java?

Question

I have been playing around with Hadoop and right now I am trying to figure out a way to read multiple files from a directory. The code below works fine when I read a file. What would be the best way to read multiple files from hdfs and read each line form every file.

try {
        Path pt = new Path("hdfs://profile/generate/work/output/errors.txt");
        FileSystem fs = FileSystem.get(job.getConfiguration());
        BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(pt)));
        String line;
        line=br.readLine();
        while (line !=null){
            //sendemail
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

You could have a great win in performance by having a thread that does the reading. Doing so you could spawn a thread for each file, hence reading multiple files at once. — KeV
– KeV, Commented Aug 10, 2015 at 15:06

TiGi · Accepted Answer · 2016-07-29 09:34:44Z

1

Just add a FileStatus[] status = fs.listStatus(new Path(path)) and make a for on your status before read one hdfs file

FileStatus[] status = fs.listStatus(new Path("path"));

for (int i=0;i<status.length;i++){

        //afficher le contenu d'un fichier hdfs
        BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(status[i].getPath())));
        String line;
        line=br.readLine();
        while (line != null){
            System.out.println(line);
            line=br.readLine();
        }
    }

answered Jul 29, 2016 at 9:34

TiGi

378 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I read multiple files from hdfs using java?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest