HDFS Command Line Append

Question

Is there any way to append to a file on HDFS from command line like copying file:

hadoop fs -copyFromLocal <localsrc> URI

Dror Harari · Accepted Answer · 2014-07-08 10:07:39Z

5

This feature is implemented in Hadoop 2.3.0 as appendToFile with a syntax like:

hdfs dfs -appendToFile localfile /user/hadoop/hadoopfile

(it was first suggested in 2009 when the HDFS Append feature was being contemplated: https://issues.apache.org/jira/browse/HADOOP-6239 )

edited Jul 8, 2014 at 10:07

Dror Harari

3,3572 gold badges31 silver badges25 bronze badges

answered Nov 14, 2012 at 9:52

metdos

14k17 gold badges79 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

octo · Accepted Answer · 2012-11-13 21:04:33Z

1

cli doesn't support append, but httpfs and fuse both has support for appending files.

w301% ls -lA
total 0
-rw-r--r-- 1 hdfs supergroup 0 Nov 14 01:02 test.me
w301% cat test.me
w301% sync
w301% cat test.me
hello
w301% echo "more rows" >> test.me
w301% sync
w301% cat test.me 
hello
more rows

EDIT: but keep in mind, that only one client can append to file.

answered Nov 13, 2012 at 21:04

octo

6753 silver badges8 bronze badges

3 Comments

metdos Over a year ago

Do you know any reference to clear instructions to set up httpfs or fuse and try out append, I tried httpfs, but I failed.

octo Over a year ago

i meant httpfs in hdfs 2.0 hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/…

octo Over a year ago

in case of fuse, you simply open file with append flag (as usual in you favorite language or use >> in shell) and write to file. recent fuse should correctly handle this flag

Collectives™ on Stack Overflow

HDFS Command Line Append

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related