How to convert a multiple-line variable containing JSON to single line in shell?

Question

I have below variable in shell

{
  "type": "service_account",
  "project_id": "projectid234",
  "private_key_id": "aasdadsxzce5",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1234242342341",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
}

Like below:

export var1='{
      "type": "service_account",
      "project_id": "projectid234",
      "private_key_id": "aasdadsxzce5",
      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
      "client_email": "[email protected]",
      "client_id": "1234242342341",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
    }'

Now I want to convert it to a single line, what should I do? I tried various solutions but they didn't work. for instance I used below code which I found but after converting to base64 and decoding it, it just return the first line. would you mind help me?

$((echo $var1 | tr -d '\n') | base64)

echo $var is very different than echo "$var". Indeed, echo $var1 is a reasonable(arguable) way to discard all the newlines. — William Pursell
– William Pursell, Commented Oct 7, 2022 at 15:43
I had to do a double-take on this - this was my first thought as well, but then I thought "Won't echoing it without quotes parse out the escape codes?" But no, they are part of the data, just like the double-quote characters themselves, or like an embedded dollar sign would be. I'm always a little wary of unquoted vars, but this looks like maybe a prime example of when they are exactly what you want. I'm still looking for holes in that, but I think William Pursell is right. :) — Paul Hodges
– Paul Hodges, Commented Oct 7, 2022 at 16:18

0stone0 · Accepted Answer · 2022-10-07 16:16:37Z

5

Since you're dealing with JSON, I'd recommend a tool like jq, which provides us with the --compact-output option and is capable of encoding to Base64 using @base64

To single-line output

jq --compact-output '' <<< "$test"

To Base64 single-line output

jq --compact-output  '@base64' <<< "$test"

To Base64 single-line output, and decoding with bash

jq --compact-output --raw-output '@base64' <<< "$test" | base64 --decode

Note the --raw-output to remove the ""'s.

Local shell example of above lines:

$ test='{
  "type": "service_account",
  "project_id": "projectid234",
  "private_key_id": "aasdadsxzce5",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "1234242342341",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
}'
$
$ jq --compact-output '' <<< "$test"
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}
$
$
$ jq --compact-output '@base64' <<< "$test"
"eyJ0eXBlIjoic2VydmljZV9hY2NvdW50IiwicHJvamVjdF9pZCI6InByb2plY3RpZDIzNCIsInByaXZhdGVfa2V5X2lkIjoiYWFzZGFkc3h6Y2U1IiwicHJpdmF0ZV9rZXkiOiItLS0tLUJFR0lOIFBSSVZBVEUgS0VZLS0tLS1cbk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRQ2lXakE1K3hGNmdzR0Ncbk9rbklMOURPQ1FwVjJMUlhZUEVLbDUrbVh1TjQ1dndxaDZRdkc0bHcvSGk3RUpodEFuK0ZReTcreU9RWXJ3M2xcblEyQ3B4RG90VCtQVDJPdVE2TFZiYy9GK1NibFBsckszQis4YUVNbzU3UForZ253TWNRNytvZlBuekM2MzV1VVBcbnBPRzBpZE1US1xuLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLVxuIiwiY2xpZW50X2VtYWlsIjoiY2xpbmV0bWFpbEBhc2Rhc2QzNDM1LmlhbS5nc2VydmljZWFjY291bnQuY29tIiwiY2xpZW50X2lkIjoiMTIzNDI0MjM0MjM0MSIsImF1dGhfdXJpIjoiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tL28vb2F1dGgyL2F1dGgiLCJ0b2tlbl91cmkiOiJodHRwczovL29hdXRoMi5nb29nbGVhcGlzLmNvbS90b2tlbiIsImF1dGhfcHJvdmlkZXJfeDUwOV9jZXJ0X3VybCI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL29hdXRoMi92MS9jZXJ0cyIsImNsaWVudF94NTA5X2NlcnRfdXJsIjoiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vcm9ib3QvdjEvbWV0YWRhdGEveDUwOS9hc3NhZGphc2hkLmlhbS5nc2VydmljZWFjY291bnQuY29tIn0="
$
$
$ jq --compact-output --raw-output '@base64' <<< "$test" | base64 --decode
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}%
$
$

edited Oct 7, 2022 at 16:16

answered Oct 7, 2022 at 15:24

0stone0

45.5k6 gold badges53 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Amir Over a year ago

can u convert the output to base64 bit and again decode it? when I do it, I am getting only the first line!

Jetchisel Over a year ago

@Amir, the file might have windows line endings aka carriage returns.

0stone0 Over a year ago

Added another example @Amir!

0stone0 Over a year ago

Add --raw-output to the jq command. See the note in my answer.

Paul Hodges Over a year ago

yeah, jq is almost certainly your best option.

|

markp-fuso · Accepted Answer · 2022-10-07 16:09:18Z

Not sure I understand the use of base64 but a simple tr should work, eg:

$ var2=$(echo ${var1} | tr -d '\n')              # ${var1} is not wrapped in double quotes so repetitive white space is squeezed to single spaces

$ echo "${var2}"
{ "type": "service_account", "project_id": "projectid234", "private_key_id": "aasdadsxzce5", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "1234242342341", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com" }

Or as William Pursell pointed out in comments, don't wrap ${var1} in quotes and you can also eliminate the od call:

$ var2=$(echo ${var1})                           # the unquoted ${var1} also removes linefeeds so no need for tr

$ echo "${var2}"
{ "type": "service_account", "project_id": "projectid234", "private_key_id": "aasdadsxzce5", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "1234242342341", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com" }

Or if you wish to maintain all the extra white space you can wrap ${var1} in double quotes before passing to tr, eg:

$ var2=$(echo "${var1}" | tr -d '\n')            # ${var1} is wrapped in double quotes to maintain all white space

$ echo "${var2}"
{      "type": "service_account",      "project_id": "projectid234",      "private_key_id": "aasdadsxzce5",      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",      "client_email": "[email protected]",      "client_id": "1234242342341",      "auth_uri": "https://accounts.google.com/o/oauth2/auth",      "token_uri": "https://oauth2.googleapis.com/token",      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"    }

I tried to use ur solution but after convert it to base64 and decode it, only the first line return $ export varwrpped=$(echo "${var2}" | base64) $ echo $varwrpped | base64 -d { "type": "service_account", "project_id": "projectid234"base64: invalid input

Paul Hodges · Accepted Answer · 2022-10-10 13:18:21Z

Just as a quick pass, you really should be able to just do this with built-in parameter expansion in bash.

The simplest:

$: echo "$var1" # this should present it exactly as you saved it
{
      "type": "service_account",
      "project_id": "projectid234",
      "private_key_id": "aasdadsxzce5",
      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",
      "client_email": "[email protected]",
      "client_id": "1234242342341",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"
    }

$: echo "${var1//$'\n'/}" # this should strip out the newlines
{      "type": "service_account",      "project_id": "projectid234",      "private_key_id": "aasdadsxzce5",      "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n",      "client_email": "[email protected]",      "client_id": "1234242342341",      "auth_uri": "https://accounts.google.com/o/oauth2/auth",      "token_uri": "https://oauth2.googleapis.com/token",      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"    }

Assuming you also want to squeeze out extraneous whitespace with a more complex pattern, you might be tempted to use a regex...which won't work, because the parameter parsing actually usesw globbing, but regular expressions per se... but try shopt for extended globbing -

$: shopt -s extglob # c.f. https://mywiki.wooledge.org/glob#extglob
$: echo "${var1//$'\n'+( )/}" # remove a newline followed by one or more spaces
{"type": "service_account","project_id": "projectid234","private_key_id": "aasdadsxzce5","private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----END PRIVATE KEY-----\n","client_email": "[email protected]","client_id": "1234242342341","auth_uri": "https://accounts.google.com/o/oauth2/auth","token_uri": "https://oauth2.googleapis.com/token","auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

edit

There IS whitespace: this changes -----BEGIN PRIVATE KEY----- to -----BEGINPRIVATEKEY-----, I just missed it.

Since you seem to have no whitespaces embedded in your data (that I noticed), you could take out the requirement for a newline in front of them, and just remove ALL spaces to compact it even more.

$: echo "${var1//*($'\n')+( )/}"
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGINPRIVATEKEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----ENDPRIVATEKEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

Again, this changes -----BEGIN PRIVATE KEY----- to -----BEGINPRIVATEKEY-----

So...in this case, with no embedded spaces, you can achieve the best result without extended globbing at all, by just using POSIX character classes to remove all whitespace, as at the top. :)

$: echo "${var1//[[:space:]]/}" # // means global replacement
{"type":"service_account","project_id":"projectid234","private_key_id":"aasdadsxzce5","private_key":"-----BEGINPRIVATEKEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCiWjA5+xF6gsGC\nOknIL9DOCQpV2LRXYPEKl5+mXuN45vwqh6QvG4lw/Hi7EJhtAn+FQy7+yOQYrw3l\nQ2CpxDotT+PT2OuQ6LVbc/F+SblPlrK3B+8aEMo57PZ+gnwMcQ7+ofPnzC635uUP\npOG0idMTK\n-----ENDPRIVATEKEY-----\n","client_email":"[email protected]","client_id":"1234242342341","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/assadjashd.iam.gserviceaccount.com"}

There are embedded whitespaces: -----BEGIN PRIVATE KEY-----
Those are not embedded whitespace - they are embedded escapes that decode into whitespace. They are preserved in the output.
It's really not important because your answer is very clear and helpful, but yes there are "embedded whitespaces" in the input, which are not preserved:) This makes your two last proposals not usable in this case (as you explain yourself, and that's why it's not important). Indeed, BEGIN PRIVATE KEY is transformed to BEGINPRIVATEKEY :)
OMG - LOL! I scanned the key, and missed the headers, even after you explicitly showed me the spaces, lol... Thanks. You are absolutely right.

Collectives™ on Stack Overflow

How to convert a multiple-line variable containing JSON to single line in shell?

3 Answers 3

8 Comments

2 Comments

edit

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

8 Comments

2 Comments

edit

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related