Bashscript upload files to hdfs

Question

I am trying to create a bashscript to upload files from the local edge node filesystem to hdfs. I was wondering a good way to add the timestamp in the file. Having some problems with getting timestamp to work.

#!/bin/bash
echo Running upload script to hdfs...
timestamp(){date +"%T"}

hdfs dfs -put /home/myname/folder1/* /user/myname/example_1_$(timestamp).txt
hdfs dfs -put /home/myname/folder2/* /user/myname/example_2_$(timestamp).txt

Are you expecting the result pattern to be example_1_11:20:15.txt? — franklinsijo
– franklinsijo, Commented Mar 24, 2017 at 18:13

franklinsijo · Accepted Answer · 2017-03-25 14:40:22Z

2

Using date +%T is not possible as the command result would contain : characters in it like 11:12:45, and creating filenames with : character is not possible in HDFS. See Hadoop-3275.

Try this command in the script,

hdfs dfs -put /home/myname/folder1/* /user/myname/example_1_`date +%H%M%S`.txt

This will create filename like /user/myname/example_1_111245.txt.

edited Mar 25, 2017 at 14:40

answered Mar 24, 2017 at 18:35

franklinsijo

18.4k4 gold badges50 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Bashscript upload files to hdfs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related