Reading several variables from a single command's multi-line output

Question

I'm running into an issue with the read command. I'm trying to get read to run through the output of my awscli command to extract VPC_ID, VPC_CIDR and VPC_NAME. Unfortunately its no longer working since upgrading to bash 5.0.

Here is the following code: read VPC_ID VPC_CIDR VPC_NAME <<<$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=${AWS_PROFILE}-vpc" --output json | jq -r '.Vpcs[] | .VpcId,.CidrBlock, (.Tags[]|select(.Key=="Name")|.Value)')

When I run aws ec2 describe-vpcs --filters "Name=tag:Name,Values=${AWS_PROFILE}-vpc" --output json | jq -r '.Vpcs[] | .VpcId,.CidrBlock, (.Tags[]|select(.Key=="Name")|.Value)'

I get my expected output but when I attached the read command in front of it, I'm only able to assign the first variable none of the other ones...

Can you build a reproducer that works without needing an AWS account? — Charles Duffy
– Charles Duffy, Commented Apr 23, 2021 at 21:35
That said, are these outputs all on the same line? If it's a separate line per variable you need to change your read a bit. Showing the output from the working command would let folks test on their own. — Charles Duffy
– Charles Duffy, Commented Apr 23, 2021 at 21:35
BTW, note that all-caps variable names are in space reserved for variables that can modify the behavior of the shell and other POSIX-specified tools. Well-behaved scripts use at least one lowercase character in names for variables they define themselves. — Charles Duffy
– Charles Duffy, Commented Apr 23, 2021 at 21:44
See pubs.opengroup.org/onlinepubs/9699919799/basedefs/… -- keeping in mind that environment variables and regular shell variables share a namespace (since setting the latter can override the former if the names collide). To quote: The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities. — Charles Duffy
– Charles Duffy, Commented Apr 23, 2021 at 21:45

Charles Duffy · Accepted Answer · 2021-04-23 21:42:56Z

2

The problem is that read, by default, stops at the first newline it sees. (This can be overridden with the -d argument). You can work around this by running a separate read per variable, or running read with a different character used as the record delimiter.

The first approach:

{ read -r VPC_ID && read -r VPC_CIDR && read -r VPC_NAME; } < <(
  aws ec2 describe-vpcs --filters "Name=tag:Name,Values=${AWS_PROFILE}-vpc" --output json \
  | jq -r '.Vpcs[] | .VpcId,.CidrBlock, (.Tags[]|select(.Key=="Name")|.Value)'
)

The second approach, which adds a printf '\0' when the inner command is successful, which read -d '' recognizes to mean the record is complete:

IFS=$'\n' read -r -d '' VPC_ID VPC_CIDR VPC_NAME < <(
  aws ec2 describe-vpcs --filters "Name=tag:Name,Values=${AWS_PROFILE}-vpc" --output json \
  | jq -r '.Vpcs[] | .VpcId,.CidrBlock, (.Tags[]|select(.Key=="Name")|.Value)' \
  && printf '\0'
)

answered Apr 23, 2021 at 21:42

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

TravelingLex Over a year ago

This is it! Thank you!! Upgrading to Bash 5.0 screwed up my original script.

Charles Duffy Over a year ago

The only reason an older version of bash would have worked would be a bug causing newlines to be changed to spaces inside of unquoted herestrings. Bash fixed the bug, so your things that were originally separate lines now stay separate lines. If you want it to work the same way on all versions, change <<<$(...) to <<<"$(...)" and you'll see the current behavior with older shells as well.

Glenn Slaven · Accepted Answer · 2024-01-17 00:27:12Z

0

With modern bash and the AWS CLI v2 you can simplify this and you don't need jq.

read -r VPC_ID VPC_CIDR VPC_NAME< <(aws ec2 describe-vpcs \
  --filters "Name=tag:Name,Values=${AWS_PROFILE}-vpc"  \
  --query "Vpcs[*].[VpcId,CidrBlock,Tags[?Key == 'Name']|[0].Value]" \
  --output text)

answered Jan 17, 2024 at 0:27

Glenn Slaven

34.3k27 gold badges118 silver badges166 bronze badges

Collectives™ on Stack Overflow

Reading several variables from a single command's multi-line output

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related