MultiThreading in Bash Script

Question

i run php from bash, and looping as much line in text file.

this is my code

#!/bin/bash
cat data.txt | while read line
do
   echo 'scrape: '$line
   php index.php $line >> output.csv
done

how i can run Simultaneously, say 10 thread per run.

bash doesn't really have threads, but you can run processes in the background by putting & at the end of the line, or you could look at a tool like GNU parallel — Eric Renouf
– Eric Renouf, Commented Jul 19, 2017 at 0:12
You're really pushing the limits of bash. You would have a much easier time using a proper scripting language like python or ruby for this — Alexander
– Alexander, Commented Jul 19, 2017 at 0:21

that other guy · Accepted Answer · 2017-07-19 04:00:01Z

3

You can easily do this with sem from GNU parallel:

#!/bin/bash
cat data.txt | while read line
do
   echo 'scrape: '$line
   sem -j 10 php index.php $line >> output.csv
done

However, it's up to you to ensure that the output makes sense when written in parallel to the same file. You may want to write to different files and join them afterwards.

answered Jul 19, 2017 at 4:00

that other guy

125k12 gold badges187 silver badges214 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

that other guy Over a year ago

Change output.csv to something likeoutput$((i++)).csv

Ole Tange · Accepted Answer · 2017-07-23 16:04:51Z

1

While that other guy's answer is correct, sem is quite slow (300 ms per job), and can in this case it can be replaced by a single call to GNU Parallel (which takes around 300 ms in startup and 10 ms per job):

parallel -j10 "echo scrape: {}; php index.php {}" :::: data.txt > output.csv

For one file per job:

parallel -j10 "(echo scrape: {}; php index.php {}) > output{#}.csv" :::: data.txt

answered Jul 23, 2017 at 16:04

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

Collectives™ on Stack Overflow

MultiThreading in Bash Script

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related