Setting additional classpath for a hadoop tool

Question

How can I dynamically set up an additional classpath so that jars are accessible from a hadoop tool class? I'm running my job via "hadoop jar". I need something like the "-libjars" switch which unfortunately works only for mappers and reducers, and my Tool implementations does not see jars provided by "libjars"

export "HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/my.jar && hadoop jar ..." seems making "my.jar" accessible to the Tool, but then libraries that are inside the job jar in the "/lib" folder become unaccessible.

Vyacheslav · Accepted Answer · 2013-07-14 20:14:28Z

2

Additionally, if you have conflicts with some jars included into a Hadoop distribution, you can export HADOOP_USER_CLASSPATH_FIRST=true. Maybe will be useful to somebody.

answered Jul 14, 2013 at 20:14

Vyacheslav

1,2663 gold badges16 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Setting additional classpath for a hadoop tool

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related