Remove substring from a json retrieved in a bash script

Question

I do not have experience working with bash, and I have to update a small script to remove a substring from all the values in a JSON with a common pattern /development/dev/. I get something like this from GetParametersByPath (AWS service):

{
    "Parameters": [
        {
            "Name": "/development/dev/var1",
            "Type": "String",
            "Value": "Saanvi Sarkar",
            "Version": 1
        },
        {
            "Name": "/development/dev/var2",
            "Type": "String",
            "Value": "Zhang Wei",
            "Version": 1
        },
        {
            "Name": "/development/dev/var3",
            "Type": "String",
            "Value": "Alejandro Rosalez",
            "Version": 1
        },
        
    ]
}

I wanna remove the substring "/development/dev/" for all the Name values.

This is what I have at the moment

  // parameter_store_path has the value "/development/dev/"
 jq_actions=$(echo -e ".Parameters | .[] | [.Name, .Value] | \042\(.[0])=\(.[2])\042 | sub(\042${parameter_store_path}/\042; \042\042)")

// function that returns the JSON 
  aws ssm get-parameters-by-path \
  --path $parameter_store_path \
  --with-decryption \
  --region eu-west-2 \
  | jq -r "$jq_actions" >> /opt/elasticbeanstalk/deployment/custom_env_var

  cp /opt/elasticbeanstalk/deployment/custom_env_var /opt/elasticbeanstalk/deployment/env

  #Remove temporary working file.
  rm -f /opt/elasticbeanstalk/deployment/custom_env_var

  #Remove duplicate files upon deployment.
  rm -f /opt/elasticbeanstalk/deployment/*.bak

I was reading some docs about bash, and I saw that I can replace part of the string using ${FOO#prefix}, but I don't know how to implement it in this code.

tripleee · Accepted Answer · 2022-10-29 20:07:33Z

2

You already seem to know how to perform substitutions in jq. Just do slightly more.

aws ssm get-parameters-by-path \
  --path "$parameter_store_path" \
  --with-decryption \
  --region eu-west-2 |
jq -r '.Parameters | .[] |
  "\(.Name | sub("/development/dev/"; ""))=\(.Value)"'

(The subscript [2] didn't make sense here - I'm guessing you meant [1]. However, the subscripting seemed superfluous, anyway; I refactored to simplify it away, and apply the substitution only on the Name. Perhaps also add double quotes around the Value?)

Demo: https://jqplay.org/s/o7Exg4Ns6BU

If you want to pass in the path as a variable, try

jq --arg path "/development/dev/" \
   -r '.Parameters | .[] |
  "\(.Name | sub($path; ""))=\"\(.Value)\""'

Demo: https://replit.com/@tripleee/aws-jq-demo#main.sh

This also gets rid of the useless use of echo -e. In this particular case, I simply switched to single quotes instead of double (though notice that they do slightly different things - generally prefer single quotes, unless you need to perform command substitution or variable substitution in your string, or if it needs to contain literal single quotes. Bash also provides $'\047C-style\047 strings' with a backslash interpretation facility similar to the one in echo -e - the example demonstrates how to embed literal single quotes with the octal escape \047).

A parameter expansion only really makes sense when what you have is already a string in a Bash variable.

Here's a refactoring of your script which also avoids the weird cp + rm in favor of a simple mv. I'm guessing you are writing something else to the target file earlier in the script and that's why you append these values from jq.

#!/bin/bash
parameter_store_path="/development/dev/"

aws ssm get-parameters-by-path \
  --path "$parameter_store_path" \
  --with-decryption \
  --region eu-west-2 |
jq --arg path "$parameter_store_path" \
   -r '.Parameters | .[] |
  "\(.Name | sub($path; ""))=\"\(.Value)\""
  ' >> /opt/elasticbeanstalk/deployment/custom_env_var

mv /opt/elasticbeanstalk/deployment/custom_env_var /opt/elasticbeanstalk/deployment/env

rm -f /opt/elasticbeanstalk/deployment/*.bak

edited Oct 29, 2022 at 20:07

answered Oct 29, 2022 at 10:16

tripleee

192k37 gold badges318 silver badges367 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

R0bertinski Over a year ago

I think your solution is quite close, but the result is not 100% correct, because the env variables file should not has "" var1=Saanvi Sarkar (this is correct). About this "You already seem to know how to perform substitutions in jq. Just do slightly more" Not too much, I just learned today, I am learning bash. Thanks for your help.

tripleee Over a year ago

You did not explain what you expect; but the fix is easy - I updated to add the -r option like in your original. (The output without any quoting is slightly jarring; it's neither valid Bash nor valid JSON.)

tripleee Over a year ago

The recipe to add double quotes around the value is just a quick hack, and won't work correctly if the actual value contains literal double quotes.

R0bertinski Over a year ago

The values never will have quotes, they are sanitized when they are added in the SSM parameter store in AWS , then this is not a thing to be worry I guess. Thanks you very much , i will check when back to the computer. 🙏

R0bertinski Over a year ago

Well something still not correct, I am trying to add your code to the script but possible I am doing something wong. jq_actions=$(echo -e ".Parameters | .[] | [.Name, .Value] | sub(\042${parameter_store_path}/\042; \042\042)") I got this error: "jq: error (at <stdin>:1030): array (["/development/dev) cannot be matched, as it is not a string". I do not know why the error says "array" .

|

Collectives™ on Stack Overflow

Remove substring from a json retrieved in a bash script

1 Answer 1

13 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

13 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related