bash script optimization

Question

I have this script below that will be daemonized and triggered possibly hundreds if not thousands of times by different users.

The script uses inotifywait to watch a folder for an upload and then move the uploaded file to its final destination for presentation, after rotating (backup/move) previous uploads. The code will be run against different upload folders that are created on the fly.

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do

        for dirnum in $(cd "$s3"; ls */*.png | sed 's%/.*%%' | sort -nr)
        do
                 next=$(($dirnum + 1));                       
                 mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png";
        done

        mv "$db"/"$file"  "$s3"/1/post1.png

done

What can I do to optimize it? Or should it be rewritten a faster programming language? Also, how can I test scripts under a certain amount of load?

If it is fast enough, it doesn't need to be re-written. You can test it by throwing files at it. An obvious optimization is to keep track of the count in each directory (WTF is a "folder"?) instead of computing it each time. — William Pursell
– William Pursell, Commented Oct 30, 2012 at 4:02
Related to, but an extension of, What's the correct way to loop this? — Jonathan Leffler
– Jonathan Leffler, Commented Oct 30, 2012 at 5:53

William Pursell · Accepted Answer · 2012-10-30 04:48:15Z

1

This does not give identical behavior, but it avoids sorting:

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do
    dirnum=1
    while test -e "$s3/$dirnum"; do : $(( dirnum += 1 )); done
    while test $dirnum -gt 0; do
        next=$(( dirnum + 1 ));    
        mkdir -p $s3/$next                   
        mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png"
        : $(( dirnum -= 1 ))
    done
    mv "$db/$file" "$s3"/1/post1.png
done

If you keep track of the highest number stored to $s3, you can avoid the first loop. Doing so is slightly more fragile if other processes are creating files in $s3, but in that case you have a race condition even in this simplistic solution. It would be a lot simpler to not rename the files, but put the first file uploaded in $s3/1, and the next in $s3/2. In that case the script can be written:

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder
dirnum=1
while test -e "$s3/$dirnum"; do : $(( dirnum += 1 )); done
inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do
    mkdir -p "$s3"/$dirnum
    mv "$db/$file" "$s3"/$dirnum/post$dirnum.png
    : $(( dirnum += 1 ))
done

answered Oct 30, 2012 at 4:48

William Pursell

214k49 gold badges279 silver badges317 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sirvon Over a year ago

the first script is super faster than my original, and it does what its supposed to do, move older post to the next numbered DIRECTORY. note: the only process creating files in $S3 is this script.. there will be a $S4, $S5 etc. but where in the script does it say to start iterating at directory $S3/101?? the second script is intriguing but it puts the uploaded post in a directory $S3/107 and then the next upload in $S3/108 the newest upload must be in $S3/1 always and the next to newest in $S3/2... thanks for the optimization lessons. ill go with the first script u posted!

William Pursell Over a year ago

The first inner while loop increments dirnum up to the first non-extant directory (apparently 101 in your case).

Jonathan Leffler · Accepted Answer · 2012-10-30 05:56:41Z

1

You should avoid moving so many files around, by putting new files in new directories and leaving old files in old directories alone. You may need to reverse your presentation logic, so the newest file (biggest number) is shown, instead of post1.png every time. But you speed that up by making it do less — and you make it do less by making it leave what's already there well alone.

If something is not fast enough, one of the best ways of speeding it up is to take a step back and look at the algorithms, and see whether there is a fundamentally faster algorithm that you can use. If you're already using the optimal algorithm, then you look at the details of how to speed that up, but you can sometimes get orders of magnitude improvement by revisiting the algorithm, where tweaking may get you a mere doubling of the speed.

edited Oct 30, 2012 at 5:56

answered Oct 30, 2012 at 5:50

Jonathan Leffler

760k145 gold badges961 silver badges1.3k bronze badges

1 Comment

sirvon Over a year ago

ty for the understanding. moving things around is the bottleneck. I am now forced to make the front just as dynamic as the back! I love progress. I didn't realize I'm designing an algorithm. my eyes are open.

anubhava · Accepted Answer · 2012-10-30 04:10:01Z

0

You can refactor your script like this:

!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

while read dir ev file
do
   while read dirnum
   do
      next=$(($dirnum + 1))
      mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png"
   done < <(find "$s3" -depth 2 -name "*.png" -exec dirname {} \; | sort -unr)

  mv "$db"/"$file"  "$s3"/1/post1.png

done < <(inotifywait -m -r -e attrib "$db")

answered Oct 30, 2012 at 4:10

anubhava

790k67 gold badges603 silver badges671 bronze badges

Collectives™ on Stack Overflow

bash script optimization

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related