how to make hadoop program use configuration file which contain argument list?

Question

I am written a hadoop program, I know that I can directly pass arguments to hadoop use args[], I mean currently is like this

ToolRunner.run(new Configuration(), new RunDear(), args);

but if there are many arguments, can I make a configuration file like below and pass to hadoop? where should this file located, in local file system or hdfs?

sample_size 200
input_genotype_file /data/genotypes.txt 
input_phenotype_file /data/phenotypes.txt
output_directory /outout 
mtry 200
ntree 3000
distance 0 (e.g. 0=euclidean, 1=mehalanobis
variable_important 0 (e.g. 0=information gain, 1=permutation)
etc….

David Gruzman · Accepted Answer · 2012-05-28 07:00:01Z

1

You can put the file into distributed cache, and then pass name of the file in the configuration to Your tasks.

answered May 28, 2012 at 7:00

David Gruzman

8,0981 gold badge31 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sandeep · Accepted Answer · 2012-05-28 07:43:42Z

1

You can use conf.addResource(new Path(/path/to/local/file)). This will pass the file to each and every task.

answered May 28, 2012 at 7:43

Sandeep

5461 gold badge5 silver badges22 bronze badges

Comments

JHS · Accepted Answer · 2012-05-27 09:00:08Z

0

You can make a Wrapper Class which reads these arguments and sets these in the agrs array and then pass it.

answered May 27, 2012 at 9:00

JHS

7,8812 gold badges34 silver badges54 bronze badges

2 Comments

user974270 Over a year ago

ok, in this case the configuration file is local file system, right?

user974270 Over a year ago

what if I want to run this program in aws?

Collectives™ on Stack Overflow

how to make hadoop program use configuration file which contain argument list?

3 Answers 3

Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related