How to stop map task from setup method?

Question

I have some map class inside class of job, and I need sometimes break execution of current task (The Hadoop Map-Reduce framework spawns one map task for each InputSplit generated by the InputFormat for the job):

public static class TestJobMapper
        extends Mapper<LongWritable, Text, Text, Text> {

    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        super.setup(context);
        // here I want to check some predicate, and may be break execution of task
        // http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/mapreduce/Mapper.html
    }

    // continue....

Thomas Jungblut · Accepted Answer · 2014-07-03 15:38:02Z

2

You can break it quite easily by overriding the run() method.

In the normal code, this is implemented like this:

setup(context);
try {
  while (context.nextKeyValue()) 
    map(context.getCurrentKey(), context.getCurrentValue(), context);

} finally {
  cleanup(context);
}

What you can do is to do your setup around that:

@Override
public void run(Mapper<LongWritable, Text, Text, Text>.Context context)
        throws IOException, InterruptedException {

   if(Predicate.runMapper(context)) {
      super.run(context); // do the usual setup/map/cleanup cycle
   }
}

That way, the task directly goes into completion if your predicate tells it to. Which still has some overhead, but it is easier than to change an input format.

answered Jul 3, 2014 at 15:38

Thomas Jungblut

21k6 gold badges71 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Venkat · Accepted Answer · 2014-07-03 15:07:19Z

0

You cannot break the execution at the setup method.

However if you logic for not executing mapper on certain split is based on the split no. then you may be able to use a custom InputFormat and record reader to skip certain records/ input splits.

answered Jul 3, 2014 at 15:07

Venkat

1,8101 gold badge11 silver badges14 bronze badges

Collectives™ on Stack Overflow

How to stop map task from setup method?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related