Reading a file in Java hdfs

Question

I ran into a problem running the program on a cluster and decided to read from hdfs file in functions map and reduce. How to read line by line hdfs file and burn to read rows in ArrayList?

Using TextInputFormat the default InputSplit is a FileInputSplit and will represent a full line. What exactly is the problem you're having? — rretzbach
– rretzbach, Commented Oct 31, 2012 at 22:21

Happy3 · Accepted Answer · 2012-11-01 04:49:49Z

1

Just a code snippet for demonstration：

Path path = new Path(filePath);
FileSystem fs = path.getFileSystem(context.getConfiguration()); // context of mapper or reducer
FSDataInputStream fdsis = fs.open(path);
BufferedReader br = new BufferedReader(new InputStreamReader(fdsis));
String line = "";
ArrayList<String> lines = new ArrayList<String>();
while ((line = br.readLine()) != null) {
    lines.add(line);
}
br.close();

answered Nov 1, 2012 at 4:49

Happy3

3092 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Reading a file in Java hdfs

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related