parallel processing with arguments from file

Question

I have a 10-line file apps.txt containing information (app id, api key and secret key) of 10 applications. Fields in each line of the file are arguments to a program interacting with a server. Another file data.txt containing data as input to the program. I want the program start one instance for each line in apps.txt and split data.txt to number of applications to process. How to use GNU Parallel to do this? I tried below command but can't get the desired behavior:

cat data.txt | parallel [-N1] -j10 --pipe --no-run-if-empty --line-buffer ./program.py {1} {2} {3} :::: apps.txt

apps.txt
AppID1 API_Key1 Secret_Key1
AppID2 API_Key2 Secret_Key2
...
AppID10 API_Key10 Secret_Key10

Ole Tange · Accepted Answer · 2018-06-04 11:35:23Z

2

I interpret your question in the way that you have 10 workers and you want to distribute blocks of stdin to those.

Use GNU Parallel's slot replacement string and have an array of which the index is the information of the worker. Bash's arrays count index from 0, so subtract 1 from slot().

# Set each entry in array 'worker' to one line from apps
parset worker echo :::: apps.txt
doit() {
  workerid="$1"
  echo "do stuff on ${worker[$workerid]}"
  # Read stuff from stdin and do 'wc' on that
  wc
}
# env_parallel is needed to get $worker exported
# -j10 must be the number of lines in apps.txt
cat data.txt | env_parallel -j10 --pipe doit '{= $_=slot()-1 =}'

answered Jun 4, 2018 at 11:35

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2524314 Over a year ago

That's exactly what I need. Thanks, learn a lot from your reply.

Joy Allen · Accepted Answer · 2018-06-04 03:05:19Z

0

I don't really understand what you said about data.txt. I just know:

Split file

$ split -l 10 data.txt -d -a 4 split_file

This command will generate {split_file000, split_file001, ...} from data.txt according to number of lines (10).
Split and pass command

$ cat app.txt | xargs -n 3 ./program.py

This command is equal to:

$ ./program.py APPID1 APP_KEY1 SECRET_KEY1

$ ./program.py APPID2 APP_KEY2 SECRET_KEY2

$ ...

answered Jun 4, 2018 at 3:05

Joy Allen

4023 silver badges8 bronze badges

2 Comments

user2524314 Over a year ago

The file data.txt is a huge file, I'd like to handle it using 10 jobs without manually splitting it.

Joy Allen Over a year ago

How about cat app.txt | xagrs -n 3 ./program.py data.txt 10 ? It will execute ./program.py data.txt 10 APPIDX APP_KEYX SECRET_KEYX. So, you can get input filename from sys.argv[1], get number of process from sys.argv[2], get app options from (sys.argv[3], sys.argv[4], sys.argv[5]). The you can do your splitting job.

Collectives™ on Stack Overflow

parallel processing with arguments from file

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related