Python 2.7: Why does json.loads not convert my string to a dict correctly?

Question

I am asking an ElasticSearch database to provide me with a list of indices and their creation dates using Python 2.7 and the Requests package. The idea is to quickly calculate which indices have exceeded the retention policy and need to be put to sleep.

The request works perfectly and the results are exactly what I want. However, when I run the code below, when I try to convert the json result to a dict, the type of theDict is correct but it reports a size of 1, when there should be at least a couple dozen entries. What am I doing wrong? I have a feeling it's something really dumb but I just can't snag it! :)

import json
import requests

esEndPoint = "https://localhost:9200"
retrieveString = "/_cat/indices?h=index,creation.date.string&format=json&s=creation.date"

# Gets the current indices and their creation dates

def retrieveIndicesAndDates():
    try:
        theResult = requests.get(esEndPoint+retrieveString)
        print (theResult.content)
    except Exception as e:
        print("Unable to retrieve list of indices with creation dates.")
        print("Error: "+e)
        exit(3)
    return theResult.content

def main():
    theDict = dict(json.loads(retrieveIndicesAndDates()))
    print(type(theDict)) # Reports correct type
    print(len(theDict))  # Always outputs "1" ??
    for index, creationdate in theDict.items():
        print("Index: ",index,", Creation date: ",theDict[index])
    return

The json the call returns:

[{"index":".kibana","creation.date.string":"2017-09-14T15:01:38.611Z"},{"index":"logstash-2018.07.23","creation.date.string":"2018-07-23T00:00:01.024Z"},{"index":"cwl-2018.07.23","creation.date.string":"2018-07-23T00:00:03.877Z"},{"index":"k8s-testing-internet-2018.07.23","creation.date.string":"2018-07-23T14:19:10.024Z"},{"index":"logstash-2018.07.24","creation.date.string":"2018-07-24T00:00:01.023Z"},{"index":"k8s-testing-internet-2018.07.24","creation.date.string":"2018-07-24T00:00:01.275Z"},{"index":"cwl-2018.07.24","creation.date.string":"2018-07-24T00:00:02.157Z"},{"index":"k8s-testing-internet-2018.07.25","creation.date.string":"2018-07-25T00:00:01.022Z"},{"index":"logstash-2018.07.25","creation.date.string":"2018-07-25T00:00:01.186Z"},{"index":"cwl-2018.07.25","creation.date.string":"2018-07-25T00:00:04.012Z"},{"index":"logstash-2018.07.26","creation.date.string":"2018-07-26T00:00:01.026Z"},{"index":"k8s-testing-internet-2018.07.26","creation.date.string":"2018-07-26T00:00:01.185Z"},{"index":"cwl-2018.07.26","creation.date.string":"2018-07-26T00:00:02.587Z"},{"index":"k8s-testing-internet-2018.07.27","creation.date.string":"2018-07-27T00:00:01.027Z"},{"index":"logstash-2018.07.27","creation.date.string":"2018-07-27T00:00:01.144Z"},{"index":"cwl-2018.07.27","creation.date.string":"2018-07-27T00:00:04.485Z"},{"index":"ctl-2018.07.27","creation.date.string":"2018-07-27T09:02:09.854Z"},{"index":"cfl-2018.07.27","creation.date.string":"2018-07-27T11:12:44.681Z"},{"index":"elb-2018.07.27","creation.date.string":"2018-07-27T11:13:51.340Z"},{"index":"cfl-2018.07.24","creation.date.string":"2018-07-27T11:45:23.697Z"},{"index":"cfl-2018.07.23","creation.date.string":"2018-07-27T11:45:24.646Z"},{"index":"cfl-2018.07.25","creation.date.string":"2018-07-27T11:45:25.700Z"},{"index":"cfl-2018.07.26","creation.date.string":"2018-07-27T11:45:26.341Z"},{"index":"elb-2018.07.24","creation.date.string":"2018-07-27T11:45:27.440Z"},{"index":"elb-2018.07.25","creation.date.string":"2018-07-27T11:45:29.572Z"},{"index":"elb-2018.07.26","creation.date.string":"2018-07-27T11:45:36.170Z"},{"index":"logstash-2018.07.28","creation.date.string":"2018-07-28T00:00:01.023Z"},{"index":"k8s-testing-internet-2018.07.28","creation.date.string":"2018-07-28T00:00:01.316Z"},{"index":"cwl-2018.07.28","creation.date.string":"2018-07-28T00:00:03.945Z"},{"index":"elb-2018.07.28","creation.date.string":"2018-07-28T00:00:53.992Z"},{"index":"ctl-2018.07.28","creation.date.string":"2018-07-28T00:07:19.543Z"},{"index":"k8s-testing-internet-2018.07.29","creation.date.string":"2018-07-29T00:00:01.026Z"},{"index":"logstash-2018.07.29","creation.date.string":"2018-07-29T00:00:01.378Z"},{"index":"cwl-2018.07.29","creation.date.string":"2018-07-29T00:00:04.100Z"},{"index":"elb-2018.07.29","creation.date.string":"2018-07-29T00:00:59.241Z"},{"index":"ctl-2018.07.29","creation.date.string":"2018-07-29T00:06:44.199Z"},{"index":"logstash-2018.07.30","creation.date.string":"2018-07-30T00:00:01.024Z"},{"index":"k8s-testing-internet-2018.07.30","creation.date.string":"2018-07-30T00:00:01.179Z"},{"index":"cwl-2018.07.30","creation.date.string":"2018-07-30T00:00:04.417Z"},{"index":"elb-2018.07.30","creation.date.string":"2018-07-30T00:01:01.442Z"},{"index":"ctl-2018.07.30","creation.date.string":"2018-07-30T00:08:28.936Z"},{"index":"cfl-2018.07.30","creation.date.string":"2018-07-30T06:52:16.739Z"}]

Take the raw json string and paste it here jsonformatter.curiousconcept.com and check if it is in the format you want it to be in. — vishal
– vishal, Commented Jul 30, 2018 at 12:23
This is probably because the JSON response contains a single array with several elements for each index. Can you try to take the length of that array instead? — Val
– Val, Commented Jul 30, 2018 at 12:25
OK but many people here will not be familiar with ElasticSearch. Please just show the JSON. — Daniel Roseman
– Daniel Roseman, Commented Jul 30, 2018 at 12:33
If your JSON really is an object, you don't have to pass the return value of loads to dict, because it will already be a dict. — chepner
– chepner, Commented Jul 30, 2018 at 12:36
I have edited the question with the json the database returns. — Juan Jimenez
– Juan Jimenez, Commented Jul 30, 2018 at 12:42

MisterMiyagi · Accepted Answer · 2018-07-31 12:11:01Z

2

Your error is trying to convert a list of dicts to a dict:

theDict = dict(json.loads(retrieveIndicesAndDates()))
#         ^^^^^                                     ^

That would only work for a dict of lists. It would be redundant, though.

Just use the reply directly. Each entry is a dict with the appropriate keys:

data = json.loads(retrieveIndicesAndDates())
for entry in data:
    print("Index: ", entry["index"], ", Creation date: ", entry["creation.date.string"])

So what happens when you do convert that list to a dict? Why is there just one entry?

The dict understands three initialisation methods: keywords, mappings and iterables. A list fits the last one.

Initialisation from an iterable goes through it and expects key-value iterables as elements. If one were to do it manually, it would look like this:

def sequence2dict(sequence):
    map = {}
    for element in sequence:
        key, value = element
        map[key] = value
    return map

Notice how each element is unpacked via iteration? In the reply each element is a dict with two entries. Iteration on that yields the two keys but ignores the values.

key, value = {"index":".kibana","creation.date.string":"2017-09-14T15:01:38.611Z"}
print(key, '=>', value)  # prints "index => creation.date.string"

To the dict constructor, every element in the reply has the same key-value pair: "index" and "creation.date.string". Since keys in a dict are unique, all elements collapse to the same entry: {"index": "creation.date.string"}.

edited Jul 31, 2018 at 12:11

answered Jul 30, 2018 at 12:46

MisterMiyagi

53.4k14 gold badges131 silver badges138 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Juan Jimenez Over a year ago

I knew I was doing something stupid! LOL! This works, thank you! :) I thought I was casting the list returned from json.loads into a dict.

chepner Over a year ago

What result would you expect from casting a list to a dict? Data types are not, in general, interchangeable.

Juan Jimenez Over a year ago

Casting is how you make data types interchangeable in most sane languages. I don't think Python is an exception to this.

MisterMiyagi Over a year ago

@JuanJimenez Sequences and mappings are fundamentally not interchangeable. You could as well try and cast an integer to a dict.

Collectives™ on Stack Overflow

Python 2.7: Why does json.loads not convert my string to a dict correctly?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related