Hadoop : ClassNotFound Error at MapReduce

Question

Just to state my setup before posing the question,

Hadoop Version : 1.0.3

The default WordCount example is running fine. But when I created a new WordCount program according to this page http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html

I compiled it and jar-ed it in similar fashion as given in the tutorial. But when I ran it using :

/usr/local/hadoop$ bin/hadoop jar wordcount.jar org.myorg.WordCount ../Space/input/ ../Space/output

I got the following error,

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.myorg.WordCount$Map

The whole error log has been pasted here : http://pastebin.com/GNbsfpg3

Where did I go wrong?

Community · Accepted Answer · 2020-06-20 09:12:55Z

5

There are some clues in the error messages:

12/07/14 18:09:38 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

12/07/14 18:09:38 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

You'll need to share your driver code with us (where you create and configure the job), but it appears you are not configuring the 'job jar', that is to say the job client is not given a hint as to where your code is bundled into a jar, and hence when you run your job, the classes cannot be found when the map instances actually run.

You probably want something like

jobConf.setJarByClass(org.myorg.WordCount.class);

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jul 14, 2012 at 13:38

Chris White

30.1k4 gold badges75 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Wesam Nabki · Accepted Answer · 2014-12-17 15:54:08Z

3

I had exactly the same problem, and got it fixed by adding the following on the main code

jobConf.setJarByClass(org.myorg.WordCount.class);

here you can find the full main function Configuration conf = new Configuration();

    Job job = new Job(conf, "wordcount");
    job.setJarByClass(org.myorg.WordCount.class);

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

job.waitForCompletion(true);

answered Dec 17, 2014 at 15:54

Wesam Nabki

2,65429 silver badges24 bronze badges

Comments

Siddhartha · Accepted Answer · 2013-05-30 11:38:18Z

0

I also got the above error. What I did is that I just copied the jar files into all nodes of the cluster and set the classpath such that every slave node can access this jar. And this worked for me. It might help you.

answered May 30, 2013 at 11:38

Siddhartha

1

Comments

guest · Accepted Answer · 2013-08-19 08:19:12Z

-1

before runJob, set conf like this: conf.setJar("Your jar file name"); it may work, try it !

answered Aug 19, 2013 at 8:19

guest

1

Collectives™ on Stack Overflow

Hadoop : ClassNotFound Error at MapReduce

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related