hdfs java file system API: creating Configuration object

Question

I was trying to create a java program to write/read files from HDFS.

I saw some examples of Java API. With this, the following code works for me.

Configuration mConfiguration = new Configuration();
mConfiguration.set(“fs.default.name”, “hdfs://NAME_NODE_IP:9000″);

But my set up has to be changed for a Hadoop HA set up so hardcoded namenode addressing is not possible.

I saw some example where in we provide the path of configuration xmls like below.

mConfiguration.addResource(new Path(“/usr/local/hadoop/etc/hadoop/core-site.xml”));
mConfiguration.addResource(new Path(“/usr/local/hadoop/etc/hadoop/hdfs-site.xml”));

This code also works when running the application in the same system as of hadoop.

But it would not work when my application is not running on the same m/c as of hadoop.

So, what is the approach I should take so that the system works but direct namenode addressing is not done.

Any help would be appreciated.

arghtype · Accepted Answer · 2019-03-12 16:38:03Z

4

While using the Hadoop High Availability concept, you need to set following properties in configuration object:

Configuration conf = new Configuration(false);
conf.set("fs.defaultFS", "hdfs://nameservice1");
conf.set("fs.default.name", conf.get("fs.defaultFS"));
conf.set("dfs.nameservices","nameservice1");
conf.set("dfs.ha.namenodes.nameservice1", "namenode1,namenode2");
conf.set("dfs.namenode.rpc-address.nameservice1.namenode1","hadoopnamenode01:8020");
conf.set("dfs.namenode.rpc-address.nameservice1.namenode2", "hadoopnamenode02:8020");
conf.set("dfs.client.failover.proxy.provider.nameservice1","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

Try it out!

edited Mar 12, 2019 at 16:38

arghtype

4,54211 gold badges51 silver badges62 bronze badges

answered Mar 10, 2016 at 17:43

Nishu Tayal

20.9k8 gold badges53 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

hdfs java file system API: creating Configuration object

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related