Bash: how to traverse directory structure and execute commands?

Question

I have split a large text file into a number of sets of smaller ones for performance testing that i'm doing. There are a number of directories like this:

/home/brianly/output-02 (contains 2 files myfile.chunk.00 and myfile.chunk.01)
/home/brianly/output-04 (contains 4 files...)
/home/brianly/output-06 (contains 6 files...)

It's important to note that there is an increasing number of files in each directory. What I need to do is run an executable against each of the text files in the output directories. The command looks something like this against a single file:

./myexecutable -i /home/brianly/output-02/myfile.chunk.00 -o /home/brianly/output-02/myfile.chunk.00.processed

Here the -i parameter is the input file and -o parameter is the output location.

In C# I'd loop over the directories get the list of files in each folder, then loop over them to run the commandlines. How do I traverse a directory structure like this using bash, and execute the command with the correct parameters based on the location and files in that location?

Lars Haugseth · Accepted Answer · 2009-06-16 20:06:52Z

9

For this kind of thing I always use find together with xargs:

$ find output-* -name "*.chunk.??" | xargs -I{} ./myexecutable -i {} -o {}.processed

Now since your script processes only one file at a time, using -exec (or -execdir) directly with find, as already suggested, is just as efficient, but I'm used to using xargs, as that's generally much more efficient when feeding a command operating on many arguments at once. Thus it's a very useful tool to keep in one's utility belt, so I thought it ought to be mentioned.

answered Jun 16, 2009 at 20:06

Lars Haugseth

14.9k2 gold badges48 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

S.Lott · Accepted Answer · 2009-06-16 19:43:17Z

8

Something like:

for x in `find /home/brianonly -type f`
do
./yourexecutable -i $x -o $x.processed
done

edited Jun 16, 2009 at 19:43

S.Lott

393k83 gold badges521 silver badges791 bronze badges

answered Jun 16, 2009 at 19:41

nikudesu

2381 silver badge3 bronze badges

4 Comments

nikudesu Over a year ago

there are back-ticks before find and after f

Kevin Little Over a year ago

for x in $(find /home/brianonly -type f); do ... done $() is so much more readable than back-ticks. My $.02.

Dennis Williamson Over a year ago

I think yours would try to process the processed files.

nikudesu Over a year ago

Dennis, you're right. I am standing by my disclaimer of: something like :)

Adam Rosenfield · Accepted Answer · 2009-06-16 19:56:58Z

2

As others have suggested, use find(1):

# Find all files named 'myfile.chunk.*' but NOT named 'myfile.chunk.*.processed'
# under the directory tree rooted at base-directory, and execute a command on
# them:
find base-directory -name 'output.*' '!' -name 'output.*.processed' -exec ./myexecutable -i '{}' -o '{}'.processed ';'

answered Jun 16, 2009 at 19:56

Adam Rosenfield

403k103 gold badges524 silver badges600 bronze badges

Comments

ephemient · Accepted Answer · 2009-06-16 20:25:00Z

1

From the information provided, it sounds like this would be a completely straightforward translation of your C# idea.

for i in /home/brianly/output-*; do
    for j in "$i/"*.[0-9][0-9]; do
        ./myexecutable -i "$j" -o "$j.processed"
    done
done

answered Jun 16, 2009 at 20:25

ephemient

207k39 gold badges292 silver badges403 bronze badges

Comments

S.Lott · Accepted Answer · 2009-06-16 19:41:18Z

0

That's what the find command is for.

http://linux.die.net/man/1/find

answered Jun 16, 2009 at 19:41

S.Lott

393k83 gold badges521 silver badges791 bronze badges

Comments

Methos · Accepted Answer · 2009-06-16 19:46:58Z

0

Use find and exec. Have a look at following

http://tldp.org/LDP/abs/html/moreadv.html

answered Jun 16, 2009 at 19:46

Methos

14.6k12 gold badges50 silver badges49 bronze badges

Collectives™ on Stack Overflow

Bash: how to traverse directory structure and execute commands?

6 Answers 6

Comments

4 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

4 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related