2

I am trying to run a script on multiple lists of files while also passing arguments in parallel. I have file_list1.dat, file_list2.dat, file_list3.dat. I would like to run script.sh which accepts 3 arguments: arg1, arg2, arg3.

For one run, I would do:

sh script.sh file_list1.dat $arg1 $arg2 $arg3

I would like to run this command in parallel for all the file lists.

My attempt:

Ncores=4
ls file_list*.dat | xargs -P "$Ncores" -n 1 [sh script.sh [$arg1 $arg2 $arg3]]

This results in the error: invalid number for -P option. I think the order of this command is wrong.

My 2nd attempt:

echo $arg1 $arg2 $arg3 | xargs ls file_list*.dat | xargs -P "$Ncores" -n 1 sh script.sh

But this results in the error: xargs: ls: terminated by signal 13

Any ideas on what the proper syntax is for passing arguments to a bash script with xargs?

4
  • 2
    You seem to have a typo of not passing the variable content of Ncores in the command but just passing the literal string which is an incorrect value type for the -P flag. Do xargs -P "$Ncores" -n 1 sh script.sh Commented Feb 20, 2019 at 18:06
  • Thanks for pointing that out Inian. That was certainly a typo on my part. I've now fixed it in my question. I still run into problems as this is still the incorrect syntax . Any ideas? Commented Feb 20, 2019 at 19:43
  • 1
    Please what is the 'real' intent? It's not very clear from the code. Commented Feb 20, 2019 at 19:51
  • Thanks liborm, I've tried to make my intent clearer in my post. Commented Feb 20, 2019 at 21:14

1 Answer 1

4

I'm not sure I understand exactly what you want to do. Is it to execute something like these commands, but in parallel?

sh script.sh $arg1 $arg2 $arg3 file_list1.dat
sh script.sh $arg1 $arg2 $arg3 file_list2.dat
sh script.sh $arg1 $arg2 $arg3 file_list3.dat
...etc

If that's right, this should work:

Ncores=4
printf '%s\0' file_list*.dat | xargs -0 -P "$Ncores" -n 1 sh script.sh "$arg1" "$arg2" "$arg3"

The two major problems in your version were that you were passing "Ncores" as a literal string (rather than using $Ncores to get the value of the variable), and that you had [ ] around the command and arguments (which just isn't any relevant piece of shell syntax). I also added double-quotes around all variable references (a generally good practice), and used printf '%s\0' (and xargs -0) instead of ls.

Why did I use printf instead of ls? Because ls isn't doing anything useful here that printf or echo or whatever couldn't do as well. You may think of ls as the tool for getting lists of filenames, but in this case the wildcard expression file_list*.dat gets expanded to a list of files before the command is run; all ls would do with them is look at each one, say "yep, that's a file" to itself, then print it. echo could do the same thing with less overhead. But with either ls or echo the output can be ambiguous if any filenames contain spaces, quotes, or other funny characters. Some versions of ls attempt to "fix" this by adding quotes or something around filenames with funny characters, but that might or might not match how xargs parses its input (if it happens at all).

But printf '%s\0' is unambiguous and predictable -- it prints each string (filename in this case) followed by a NULL character, and that's exactly what xargs -0 takes as input, so there's no opportunity for confusion or misparsing.

Well, ok, there is one edge case: if there aren't any matching files, the wildcard pattern will just get passed through literally, and it'll wind up trying to run the script with the unexpanded string "file_list*.dat" as an argument. If you want to avoid this, use shopt -s nullglob before this command (and shopt -u nullglob afterward, to get back to normal mode).

Oh, and one more thing: sh script.sh isn't the best way to run scripts. Give the script a proper shebang line at the beginning (#!/bin/sh if it uses only basic shell features, #!/bin/bash or #!/usr/bin/env bash if it uses any bashisms), and run it with ./script.sh.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.