How to feed a list of files into a bash script

Question

I'm having problems feeding a large list of filenames into a command in (git for windows git, I think it's cygwin)bash shell.

This is the basic command that works fine with a small set of arguments: git filter-branch -f --tree-filter 'rm -rf file1 directory2 file3 file4' However, I have about 1500 filenames.

I've tried: git filter-branch -f --tree-filter 'rm -rf file1 directory2... all 1500 names here' but I get an error:

/mingw64/bin/git: Argument list too long

I've tried to use a for loop: git filter-branch -f --tree-filter 'for f in $(cat files.txt) ; do rm -fr "$f" ; done' and this runs through the loop with an error:

cat: files.txt: No such file or directory

FYI - the files.txt contents look like this:

./file1 
./directory2 
./file3 
./file4

Then I tried: git filter-branch -f --tree-filter < cat files.txt and cat files.txt | git filter-branch -f --tree-filter but I get errors about the syntax not being correct - it shows the 'help' dialogue. eg:

usage: git filter-branch [--setup ] [--subdirectory-filter ] [--env-filter ] [--tree-filter ] [--index-filter ] [--parent-filter ] [--msg-filter ] [--commit-filter ] [--tag-name-filter ] [--original ] [-d ] [-f | --force] [--state-branch ] [--] [...]

Then I thought maybe I could just add the arguments into the file like this: git filter-branch -f

File:

--tree-filter './file1 ./directory2 ./file3 ./file4'

But I get the 'help' dialogue again.

I'm sure there is a way to do this, but my unix-fu is too weak. Please help!

In response to @dash-o:

I tried this, but am getting an error:

C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: eval: line 414: unexpected EOF while looking for matching `'' C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: eval: line 415: syntax error: unexpected end of file

The files.txt lists the files one per line. However, the rm -rf requires them to be on a single line and space-delimited.

I tried to put the files names on a single line, but I get a different error:

C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: line 414: rm -rf .vs ./file1 ./directory2: command not found tree filter failed: 'rm -rf ./file1 ./directory2'

Maybe the single quotes are being escaped and are not wrapped around the rm command?

Possible duplicate of remove deleted files from git: Argument list too long, Argument list too long error for rm, cp, mv commands, etc. — jww
– jww, Commented Nov 7, 2019 at 15:34

nullPointer · Accepted Answer · 2019-11-07 10:43:37Z

2

Give this a try, just make sure you're in the path where files.txt is located to avoid the "No such file or directory" error :

for f in $(cat files.txt); do git filter-branch -f --tree-filter 'rm -rf '"$f"'' ; done

answered Nov 7, 2019 at 10:43

nullPointer

4,5891 gold badge20 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dash-o · Accepted Answer · 2019-11-07 13:06:09Z

1

For cases of larger number of files, it might be time consuming to iterate over each file individually. For those cases, consider using xargs ability to batch arguments together (based on max number of argument, max command line size, etc).

xargs -L50 < files.txt  | xargs -I@ git filter-branch -f --tree-filter "'rm -rf @'"

The first xargs is used purely to arrange block arguments, 50 files each. This can be customized to include a line size limit, if needed.

The second xargs will execute git filter for each 50 arguments. You might need to work on the quoting, as I do not have the ability to test the command with git on windows).

answered Nov 7, 2019 at 13:06

dash-o

14.6k1 gold badge14 silver badges43 bronze badges

2 Comments

Lanceomagnifico Over a year ago

Sorry, didn't see the last line about the quotes - I double single-quoted the rm -rf part and it seems to be working...

Lanceomagnifico Over a year ago

Working! I took the file and made it one line and altered your solution a bit: xargs -n300 < files.txt | xargs -I@ git filter-branch -f --tree-filter "''rm -rf @''" I'll mark yours as the answer (but thanks @funkjelly) as you are correct about me wanting to batch the operation to save time using xargs.

Collectives™ on Stack Overflow

How to feed a list of files into a bash script

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related