GNU parallel with nested for loops and multiple commands

Question

I am trying to run 10 instances of a BASH function simultaneously with GNU Parallel

The BASH function downloads tiles from an image and stitches them together - first single rows, then each column - to a single image file.

function DOWNLOAD_PAGE {
for PAGE in {0041..0100}
do
                for COLUMN in {0..1}
                        do
                                for ROW in {0..2}
                                do wget -O "$PAGE"_"$COLUMN"_"$ROW".jpg "http://www.webb$PAGE$COLUMN$ROW"
                        done
                convert "$PAGE"_"$COLUMN"_*.jpg -append "$PAGE"__"$COLUMN".jpg
                done
                convert "$PAGE"__*.jpg +append "$PAGE"_done.jpg
        done
}

Unfortunately, the apparently obviuous solutions - the first one being

export -f DOWNLOAD_PAGE
parallel -j10 DOWNLOAD_PAGE

do not work.

Is there a way to do this using GNU Parallel?

Ole Tange · Accepted Answer · 2014-02-11 23:31:25Z

Parts of your function can be parallized and others cannot: E.g. you cannot append the images before you have downloaded them.

function DOWNLOAD_PAGE {
    export PAGE=$1
    for COLUMN in {0..1}
    do
        parallel wget -O "$PAGE"_"$COLUMN"_{}.jpg "http://www.webb$PAGE$COLUMN{}" ::: {0..2}
        convert "$PAGE"_"$COLUMN"_*.jpg -append "$PAGE"__"$COLUMN".jpg
    done
    convert "$PAGE"__*.jpg +append "$PAGE"_done.jpg
}

export -f DOWNLOAD_PAGE
parallel -j10 DOWNLOAD_PAGE ::: {0041..0100}

A more parallelized version (but harder to read):

function DOWNLOAD_PAGE {
    export PAGE=$1
    parallel -I // --arg-sep /// parallel wget -O "$PAGE"_//_{}.jpg "http://www.webb$PAGE//{}"\; convert "$PAGE"_"//"_\*.jpg -append "$PAGE"__"//".jpg ::: {0..2} /// {0..1}
    convert "$PAGE"__*.jpg +append "$PAGE"_done.jpg
}

export -f DOWNLOAD_PAGE
parallel -j10 DOWNLOAD_PAGE ::: {0041..0100}

Your understanding of what GNU Parallel does is somewhat misguided. Consider walking though the tutorial http://www.gnu.org/software/parallel/parallel_tutorial.html and then try to understand how the examples work: n1_argument_appending">http://www.gnu.org/software/parallel/man.html#example__working_as_xargs_n1_argument_appending

Collectives™ on Stack Overflow

GNU parallel with nested for loops and multiple commands

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related