-1

My use case:

I need to remove all dev packages listed in composer.json file. Suppose I have two packages: projectx/package-nice and projecty/package-good. To remove them I need to run:

$ composer remove --dev projectx/package-nice projecty/package-good

So I build this command to extract the package list:

echo $(composer show -s | grep -E "^[a-z]+/[0-9a-z_-]+" | awk '{print $1}' | xargs)

This return the list of packages, exacly like this: projectx/package-nice projecty/package-good

So I tried to run the command below, but didn't work because bash is interpreting the return as a single string enclosed by quotes:

$ composer remove --dev $(composer show -s | grep -E "^[a-z]+/[0-9a-z_-]+" | awk '{print $1}' | xargs)

It's the same as:

$ composer remove --dev "projectx/package-nice projecty/package-good"

So, what am I doing wrong?

edit:

Note that the problem is not about the parsing. The $() is returning the expected values separated by spaces. The issue is about WHY bash is interpreting this return as a unique value.

As @MarcusMüller noted, this problem shouldn't be happening. Inside /etc I ran:

$ ls $(ls | head -n 2)

and the executed command was ls file1 file2 and not ls "file1 file2", so I don't understand why this is happening. Maybe it's because composer is just a script that is run by php, and this is interfering with something?

Thank you.

Improve this question
10
  • 1
    bash interprets the return as a single string enclosed by quotes usually it doesn't, and that becomes a problem, often. So, hm. Commented Aug 19, 2022 at 18:00
  • @MarcusMüller "usually it doesn't". Well, I updated the question to reflect this. Commented Aug 19, 2022 at 18:11
  • Can you show an example of the JSON document? I expect it to be fairly easy to parse it with e.g. jq and extract the package names into an array. Commented Aug 19, 2022 at 18:19
  • @Kusalananda The problem isn't about the json parsing, this is just an example. The problem is about executing a command that receives n parameters separated by spaces returned by the $() Commented Aug 19, 2022 at 18:23
  • 1
    1. This shell behavior depends on the value of the built-in variable IFS. If that is somehow set to the empty string, or it doesn't contain all of tab, newline, and space, the shell will not split strings into words in many situations, including this one. 2. What is the purpose of xargs here? I think you're mixing up two different paradigms of reusing the output of commands. Direct interpolation as arguments of another command is one paradigm, piping into xargs another, different paradigm. Commented Aug 19, 2022 at 19:07

2 Answers 2

Reset to default
2 
composer remove --dev $(composer show -s | grep -E "^[a-z]+/[0-9a-z_-]+" | awk '{print $1}' | xargs)

is not the same as:

composer remove --dev "projectx/package-nice projecty/package-good"

The $(cmd) part, because it is not quoted and in list context (in arguments to a simple command here) is subject to split+glob.

Unless you've modified it, $IFS (which is used for the split part) happens to contain the space character, so if cmd outputs projectx/package-nice projecty/package-good\n, that will be split into projectx/package-nice and projecty/package-good and passed as separate arguments to composer.

By the way, newline is also in the default value of $IFS, so your xargs (which I suppose is there to convert the newlines to spaces¹) is rather pointless.

Rather than using split+glob, if using the bash shell, using readarray to read the lines of some file into individual elements of an array would make more sense:

readarray -t packages < <(
  composer show -s | grep -Po '^\p{Ll}+/[\p{Ll}\d_-]\H*'
)
(( ${#packages[@]} == 0 )) ||
  composer remove --dev "${packages[@]}"

Using split+glob is also an option, but as always, when using it it's best to tune it for your exact need:

IFS=$'\n' # split on newline only
set -o noglob # disable the glob part which we don't want
packages=( $(cmd...) ) # split+glob, result assigned to an array

In your case, the output of cmd should not contain space nor tab characters, the other two characters that are in the default value of $IFS in bash, so you could leave $IFS as is.

But it might contain globs. For instance, if composer show -s output etc/p* blah blah for instance, your pipeline would output etc/p*, and if run from within /, without the set -o noglob, that etc/p* would be expanded to etc/pam.conf, etc/passwd, etc/profile...

To prevent split+glob, for the output of cmd (minus the trailing newlines that are stripped by command-substitution) to be passed as one and only one argument to the command, you use double quote:

composer remove --dev "$(cmd)"

(which only makes sense if cmd outputs only one package).

On Linux, you can see what arguments are being passed to a command using

strace -s999999 -qqfe execve the-command and its args

(or run that strace command on your shell to trace all the execve() system calls that it or any of the processes it spawns make)

For instance:

split+glob with the default value of IFS:

bash-5.0$ strace -s999999 -qqfe execve true $(echo foo; echo foo bar)
execve("/usr/bin/true", ["true", "foo", "foo", "bar"], 0x7ffe1374cc50 /* 66 vars */) = 0

Splitting on newline only:

bash-5.0$ (IFS=$'\n'; strace -s999999 -qqfe execve true $(echo foo; echo; echo foo bar))
execve("/usr/bin/true", ["true", "foo", "foo bar"], 0x7ffd16c547f8 /* 66 vars */) = 0

(note that the empty line is removed).

Effect of the glob part:

bash-5.0$ (strace -s999999 -qqfe execve true $(echo 'etc/p*'))
execve("/usr/bin/true", ["true", "etc/pam.conf", "etc/pam.d", "etc/papersize", "etc/parallel", "etc/passwd", "etc/passwd-", "etc/pcmcia", "etc/perl", "etc/php", "etc/pki", "etc/pm", "etc/pnm2ppa.conf", "etc/polkit-1", "etc/popularity-contest.conf", "etc/ppp", "etc/printcap", "etc/profile", "etc/profile.d", "etc/protocols", "etc/pulse", "etc/python2.7", "etc/python3", "etc/python3.8"], 0x7ffdc911f8a0 /* 66 vars */) = 0

Fixed with set -o noglob:

bash-5.0$ (set -o noglob; strace -s999999 -qqfe execve true $(echo 'etc/p*'))
execve("/usr/bin/true", ["true", "etc/p*"], 0x7ffe9c278a50 /* 66 vars */) = 0

Split+glob disabled by quoting:

bash-5.0$ (IFS=$'\n'; strace -s999999 -qqfe execve true "$(echo foo; echo; echo foo bar; echo 'etc/p*')")
execve("/usr/bin/true", ["true", "foo\n\nfoo bar\netc/p*"], 0x7ffcf0e70d20 /* 66 vars */) = 0

In zsh, only the IFS-splitting part is done upon unquoted command substitution, not the glob part (ksh also does brace expansion in addition to split+glob). In zsh, you can also apply an explicit splitting on top of command substitution using the s, f, 0 parameter expansion flags. For instance ${(f)"$(cmd)"} splits the output of cmd on newlines, so here, you'd do:

packages=( ${(f)"$(composer show -s | grep -Po '^\p{Ll}+/[\p{Ll}\d_-]\H*')"} )
(( $#packages == 0 )) || composer remove --dev $packages

Without having to modify $IFS nor disable globbing globally.

¹ the wrong way to do it as there are many ways that could fail for arbitrary input and is quite inefficient.

Improve this answer
-1

You have the solution on hand. Your echo command returns 2 strings separated by a space. Enclose that one in a $() command substitution (yes, they can be nested) and you get what you need:

composer remove --dev $(echo $(composer show -s | grep -E "^[a-z]+/[0-9a-z_-]+" | awk '{print $1}' | xargs))
Improve this answer
3
  • 1
    I don't how this would help. Or do pretty much anything, really. With the default IFS, the output from the inner command substitution would be split on spaces, and passed to echo as distinct words, and then echo would join them with single spaces, print them, and then that (the output from the outer command substitution) would again be split. Just one command substitution would do the same. Commented Aug 20, 2022 at 11:08
  • @ilkkachu The problem at hand is said to be a case where the result of the innermost command substitution is NOT split on spaces. But the second one, with the echo, does the splitting just fine. Commented Aug 30, 2022 at 18:32
  • It doesn't matter. An unquoted command substitution would split (or not) the same as an unquoted command substitution. That is, it'd split on whitespace if IFS is unset, or it would not if IFS is set to the empty string (or it would split on whatever IFS is set to, if it's set to something different). Perhaps their issue is that IFS is modified from the default, and there's no splitting, but that would mean there wouldn't be any splitting on the other command substitution either. So, I don't see any scenario where this would help. Commented Aug 30, 2022 at 19:51

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.