I'm trying to run the command
find . -name "*.csv" | xargs -I{} cat '{}' > Everything2.csv
and I get back:
cat: ./Everything2.csv: input file is output file
What does this mean?
3 Answers
Tell find to exclude the output file from its results to prevent this loop:
find . -name Everything2.csv -prune -o \
-name '*.csv' -exec cat {} + \
>Everything2.csv
2 Comments
Toby Speight
Looks like this would exclude all files of that name, and any directories of that name, plus their content. I think we should be using
! -path ./Everything2.csv if we want to exclude only that single file.
Charles Duffy
@TobySpeight, ...if someone names a directory
Everything2.csv, they deserve what they get. More seriously, your comment above is entirely correct, and anyone who cares about the corner case should adopt it.As shown in that answer, you should run:
$ find . -name '*.csv' -exec cat {} + | tee Everything2.csv
since redirection operator (> or >>) has a higher precedence, therefore it creating/truncating the file, before the find command is invoked. So to avoid that you need to generate the list first, then pipe it into the file, but without using redirection operator, so tee in this cause works fine.
Alternatively use sponge instead of cat which soaks up standard input and write to a file:
find . -name "*.csv" | xargs -I{} sponge '{}' > Everything2.csv
3 Comments
Charles Duffy
The suggested code works only if the output file doesn't already exist. If there's a file present whose contents we're trying to replace, the same issue can occur.
Charles Duffy
(It's not guaranteed to work if the output file doesn't exist either, because all parts of a pipeline are started in parallel; it's a race with undefined outcome whether tee creates the file before find gets to that point)
Toby Speight
"Precedence" isn't really a concept here. The true explanation (as you go on to explain) is that opening for writing via
> is done by the shell before starting the process. However, there's a problem with the solution, in that if/when find finds Everything2.csv, then cat|tee will be eating its own tail just like <x cat | cat -u >>x or dd if=x of=x bs=1 oflag=append conv=notrunc.This can be done easily using the negation operator ! as shown below:
$ find . -type f -name '*.csv' ! -name 'Everything2.csv' -exec cat {} + > Everything2.csv
-name option of the find function checks for the provided file name in the directory and the sub directories thereby avoiding reading from that file.
If you want to match with a file at a given path under the directory tree, you may use the -wholename option instead.
findis findingEverything2.csvand xargs is invokingcatwithEverything2.csvas the input file.find . -name '*.csv' -print0 | xargs -0 catis generally the safer practice, if you want to usexargs. Look at what your existing code does if some of your CSV files have spaces or quotes in their names; it's not pretty.