Multiple Output in Reducer

Question

I am working on simple map reduce program. I want to create different files after reducer for each different word in the key. For example, after executing Mapreduce I have something like

Priority1 x 2

Priority1 y 2

Priority1 z 2

priority2 x 2

priority2 y 2

Now I want different files after reduce phase, saying Priority1 and Priority2 which have all these values according to the priority. I am using java and want to know what should be written in reducer for having this kind of output?

I just want to know if this is even possible or if it is how to approach or solve this? I am using Hadoop 0.20.203 and hence multipleoutputs doesn't work.

Any pointers will be helpful. Thanks for the help! Atul

Ravi Bhatt · Accepted Answer · 2012-02-20 22:54:28Z

0

You need to create a partioner class first, that partions based on your criteria.

You then need to create your own outputformat class and a recordwriter class.

The recordwriter class, needs to write to different files as per your needs. Further if you need to sort your values create comparator class for your key field.

edited Feb 20, 2012 at 22:54

answered Feb 20, 2012 at 21:50

Ravi Bhatt

3,16321 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Huckle Over a year ago

Specifically when you create the output format, how exactly do you handle creating a new file for each word? Normally the output files are created when calling OutputFormat.getRecordWriter(context) so how would you know what to name the file?

Don Branson · Accepted Answer · 2012-02-19 20:45:39Z

0

Have a look at MultipleOutputs.

answered Feb 19, 2012 at 20:45

Don Branson

13.7k10 gold badges62 silver badges103 bronze badges

2 Comments

user722856 Over a year ago

I looked at MultipleOutputs but it is not available in hadoop 0.20.203. I apologize I forgot to mention the version of hadoop in my Question. Thanks!! Atul

Don Branson Over a year ago

ah, okay. well, i could have asked, too. :) did you see stackoverflow.com/questions/2180101/…?

Collectives™ on Stack Overflow

Multiple Output in Reducer

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related