Format of a JSON response with Python

Question

I've been working on a side-project and I've been struggling with extracting data from a JSON response using Python. Whatever I come up with, I can't seem to have a proper formatted JSON result (probably because I'm new to Python, so I could use some guidance)

With these few lines I query an API and write the response to a file:

response = client.get_products()
with open('file.json', 'w') as f:
    f.write(str(response))

The generated json file reads as follows:

Output

{
  'version': 'value',
  'list': [
    '{
        "product":
          {
            "attributes":
              {
                "name":"value",
                "location":"value"
              },
            "description":"text productA"
          },
        "version":"value"
    }',
    '{
        "product":
          {
            "attributes":
              {
                "name":"value",
                "location":"value"
              },
            "description":"text productB"
          },
        "version":"value"
    }'
  ],
 'date':'value'
}

That does not look like valid JSON, notice the single quotes... but, all within {"product":...} looks fine. So i tried another approach:

response = client.get_products()
with open('file.json', 'w') as output:
    json.dump(response, output)

This time the json file contains the following:

Output

{
  "version": "value", 
  "list": [
    "{
        \"product\":
          {
            \"attributes\":
              {
                \"name\":\"value\",
                \"location\":\"value\"
              },
            \"description\":\"text productA\"
          },
        \"version\":\"value\"
    }", 
    "{
        \"product\":
          {
            \"attributes\":
              {
                \"instanceType\":\"value\",
                \"location\":\"value\"
              },
            \"description\":\"text productB\"
          },
        \"version\":\"value\"
    }"            
  ], 
  "date": "value"
}

Hmmm, not what I had in mind either...

Question: What am I doing wrong? and more important, how can I write the response to a valid formatted JSON file? So expected outcome should be:

Expected output

{
  "version": "value",
  "list": [
    {
        "product":
          {
            "attributes":
              {
                "name":"value",
                "location":"value"
              },
            "description":"text productA"
          },
        "version":"value"
    }, 
    {
        "product":
          {
            "attributes":
              {
                "name":"value",
                "location":"value"
              },
            "description":"text productB"
          },
        "version":"value"
    }
  ], 
  "date": "value"
}

In the end I which to isolate all "product" entries in the response, bonus points for additional tips on how to achieve this :)

Is response an actual python dict? It looks like you're working with a string since it has both single and double quotes. — roganjosh
– roganjosh, Commented Jan 1, 2018 at 16:08

Martijn Pieters · Accepted Answer · 2018-01-01 16:08:43Z

2

The object contains strings with JSON encoding. Each element in the list referenced by the list key is a string object, that happens to hold valid JSON data. By using json.dumps() you encoded those strings to JSON values, so the use of backslashes is entirely correct; it is opaque data otherwise, it doesn't matter to the JSON encoder that the data is itself also valid JSON.

That those elements are JSON strings is probably an error on the side of the service you called. You now need to undo that mistake by decoding those JSON strings first:

response = client.get_products()
response['list'] = [json.loads(s) for s in response['list']]
with open('file.json', 'w') as output:
    json.dump(response, output)

If you also are responsible for the API output, fix that output. Don't double-encode your data.

As you discovered, using str() does not produce JSON output. That produces a Python representation, using valid Python syntax.

answered Jan 1, 2018 at 16:08

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

erik258 Over a year ago

get_products is likely combining multiple Json encoded strings as list elements. You will either have to double decode on the caller side , or, more logically, change within get_products to decode the individual Json and then reencode the entire thing once.

eli45 Over a year ago

@Martijn That makes sense, thank you for the explanation and adapted code, works like a charm! I'll get in touch with the owner of the API, or I should say the get_products function, as other functions provide by the API use correct encoding

Collectives™ on Stack Overflow

Format of a JSON response with Python

Output

Output

Expected output

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Output

Output

Expected output

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related