How can I run Hadoop run with a Java class?

Question

I am following the book Hadoop: the definitive Guide.

I am confused on example 3-1.

There is a Java source file, URLCat.java. I use javac to compile it into URLCat.class, then use jar to wrap it into a jar.

The book said to use

% hadoop URLCat hdfs://localhost/user/tom/quangle.txt

to run it. I have tried a lot of different ways, such as

% hadoop jar URLCat.jar .......

but didn't work. I got errors like this:

Exception in thread "main" java.lang.ClassNotFoundException: hdfs://localhost/user/username/quangle/txt

What is the reason for this, and how do I do it right?

SquareCog · Accepted Answer · 2010-09-10 08:59:17Z

4

It's quite simple:

[me@myhost ~]$ hadoop jar
RunJar jarFile [mainClass] args...

So, what you want is hadoop jar yourJar.jar your.class.with.Main [any args]

answered Sep 10, 2010 at 8:59

SquareCog

19.7k8 gold badges52 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mtk · Accepted Answer · 2012-06-25 12:45:34Z

Of course you could use cat, but that sort of isn't the point (i.e. you're learning, not just trying to get it to work).

As per the book, you need to set your HADOOP_CLASSPATH environment variable. In my case, using the build example in the book, all of my classes are at: /media/data/hadefguide/book/build/classes

Here's an example:

hduser@MuleBox ~ $ export HADOOP_CLASSPATH=
hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt
Exception in thread "main" java.lang.NoClassDefFoundError: URLCat
Caused by: java.lang.ClassNotFoundException: URLCat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: URLCat.  Program will exit.
hduser@MuleBox ~ $ export HADOOP_CLASSPATH=/media/data/hadefguide/book/build/classes
hduser@MuleBox ~ $ hadoop URLCat hdfs://localhost/user/hduser/quangle.txt
On the top of the Crumpetty Tree
The Quangle Wangle sat,
But his face you could not see,
On account of his Beaver Hat.

OneCricketeer · Accepted Answer · 2016-09-27 16:29:12Z

2

Not sure how Useful is the answer now. I faced the same issue today in fact working on an example from the same book (Hadoop definitive guide) I was able to execute an example program as follows:

Write your java code and save it as .java file

Compile your java program using:

javac -classpath <path to hadoop core and commons-cli jar file> <path to your java program file>

Create a jar file containing your class file:

jar cvf <jar file> <class files to add separated by space>

Execute the jar file using hadoop command line:

hadoop jar <jar file name> <class name containing your main method> <argument to the main method>

e.g.

hadoop jar FileSystemCat.jar FileSystemCat hdfs://localhost/user/root/MyFiles/meet_a_seer.txt

Hope it helps

edited Sep 27, 2016 at 16:29

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

answered Dec 1, 2014 at 7:28

Punit Sachdev

311 bronze badge

Comments

khmarbaise · Accepted Answer · 2010-08-31 07:38:58Z

0

The syntax of the command is a little bit different:

hadoop fs -cat hdfs:///user/tom/quangle.txt

Do you have hadoop home in your path? can you call hadoop without any parameters?

answered Aug 31, 2010 at 7:38

khmarbaise

98.4k28 gold badges198 silver badges254 bronze badges

2 Comments

Josh Morrison Over a year ago

re syntax, I have and I can. No problem.

Tim Goodman Over a year ago

Why is this the accepted answer? The whole point of the example is to learn how to write and run a Java program that reads from hdfs.

Dmytro Molkov · Accepted Answer · 2010-09-03 02:03:08Z

0

To make the hadoop URLCat command work you need to get the jar (URLCat.jar) to be in your class path. You can put it in lib/ dir of hadoop for that.

For the hadoop jar URLCat.jar to run you need to create a jar that will have Main class defined in it, otherwise it thinks that the next argument on the command line is the class name. What you can try is hadoop jar URLCat.jar URLCat hdfs://...

answered Sep 3, 2010 at 2:03

Dmytro Molkov

111 bronze badge

Comments

Balayesu Chilakalapudi · Accepted Answer · 2015-03-12 10:37:08Z

0

I did this based on help found on this site and the hadoop tutorial.

mkdir urlcat_classes<br>
javac -classpath /usr/lib/hadoop/hadoop-0.20.2-cdh3u1-core.jar -d     urlcat_classes URLCat.java<br>
jar -cvf urlcat.jar -C urlcat_classes .<br>
hadoop jar urlcat.jar no.gnome.URLCat       
hdfs://localhost/user/claus/sample.txt<br>
<br>
no.gnome is from 'package no.gnome;' in URLCat.java.<br><br>

regards
Claus

edited Mar 12, 2015 at 10:37

Balayesu Chilakalapudi

1,4023 gold badges20 silver badges43 bronze badges

answered Oct 1, 2011 at 20:57

kometen

8,1406 gold badges48 silver badges64 bronze badges

Comments

Kishore Bhosale · Accepted Answer · 2016-01-12 05:47:16Z

0

step 1: Compile Java Program:

javac URLCat.java -classpath $HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.0.jar

step 2: Create jar file :

jar cvf URLCat.jar URLCat.class

Step 3: Execute program : (mention your hdfs file location)

hadoop jar URLCat.jar URLCat hdfs://localhost:9000/pcode/wcinput.txt

answered Jan 12, 2016 at 5:47

Kishore Bhosale

5695 silver badges7 bronze badges

Comments

Community · Accepted Answer · 2017-05-23 11:45:22Z

0

Go to the directory where your compiled .class files are residing.

Use full class name including package name (refer to Receiving "wrong name" NoClassDefFoundError when executing a Java program from the command-line for full class name or which directory to run the job in) when running hadoop URLCat hdfs://localhost/user/tom/quangle.txt.

In my case URLCat.java was in com.tom.app, so the hadoop command was hadoop com.tom.app.URLCat hdfs://localhost/user/tom/quangle.txt.

edited May 23, 2017 at 11:45

CommunityBot

11 silver badge

answered Jan 19, 2016 at 7:05

Eric

2,7257 gold badges32 silver badges70 bronze badges

Comments

Apurv · Accepted Answer · 2012-01-12 04:24:32Z

-2

We can access HDFS through the hdfs api. My understanding of it is that you can use the hdfs api to contact a hadoop cluster running the dfs and fetch data from it.

Why do we need to invoke the command as hadoop jar URLCat.jar

why not just java URLCat

Why does the client necessarily need to install hadoop and then contact the hadoop cluster?

answered Jan 12, 2012 at 4:24

Apurv

4,5332 gold badges24 silver badges33 bronze badges

Collectives™ on Stack Overflow

How can I run Hadoop run with a Java class?

9 Answers 9

Comments

Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

Comments

Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related