How to Create a Dynamic Path for Grabbing Variables from JSON in Python

Question

I'm calling many web services in Python which return JSON formatted text describing the same data, but each are structured differently. Rather than write a dozen ways to loop through the data, I'm attempting to create a common base path and common field name, but not understanding now to create a dynamic name to grab the desired field.

service1 calls the street address, "addr1" while service2 calls it, "address1".

service1 JSON path to the street address: ["results"][0]["locations"][0]["addr1"] .

service2 JSON path to the street address: ["query"][0]["address1"] .

service1 response:

{
    "results":[
        {
            "providedLocation":{
                "location":"123 Main St. Whoville"
            },
            "locations":[
                {
                "addr1":"123 Main St.",
                "city":"Whoville"
                }
            ]
        }
    ]
}

From my common terms translation.ini file

[service1]
base_path = '["results"][0]["locations"]' 
field     = "addr1"
[service2]
base_path = '["query"]'
field     = 'address1'

I'm looping through the address locations returned like this:

j         = json.loads(response.text)
try:
    for i in range(0,count_the_results()):
        street_number = j[base_path][i][field]  # multiple locations returned

except KeyError:
    print("street_number error again")

I tried using eval(), but it drops the brackets ([0]) which breaks the path

j[eval(base_path)]

I tried breaking the base_path into pieces but eval still gets me

base_path = '["results"][0]'
locations = '["locations"]'
j[eval(str(base_path)+str(locations))]

I could use keys, but still have to build the unique path somehow

j.get("results",{})[0].get("locations")[i][field]

Could someone point out what I'm missing in creating the path, please?

Is there any reason you are always using the first result of each list? Both service models in your examples support having several addresses; shouldn't you code in a way that you can handle them all? — Joe
– Joe, Commented Mar 24, 2016 at 23:12
good question, Joe. The "base_path" is the part I've identified as a static part of the response - always 1 item, and it just happens to be named 'results' (plural). This is followed by an unknown list number of addresses, incremented by [i] in my example, before the desired field. Thank you for your reply. — jim
– jim, Commented Mar 24, 2016 at 23:55
you can always do a step through method where you look at the keys if they are a certain string. this way you only need to have the name of the variable. — Headhunter Xamd
– Headhunter Xamd, Commented Mar 25, 2016 at 9:13

Ilya Peterov · Accepted Answer · 2016-03-25 08:59:30Z

4

You could do it like that:

def get_item(response, path):
    for item in path:
        response = response[item]
    return response

# Assume we need the string 'qwerty'
spam = {'spam': ['eggs', {'13': 'qwerty'}], 'eggs': 123}
# Then that will be our path:
path = ['spam', 1, '13']

print get_item(spam, path)

It prints 'qwerty'.

The shorted and more efficient (but probably less understandable) way of doing this is to use builtin reduce function like that:

def get_item(response, path):
    return reduce(lambda x, y: x[y], path, response)

The other problem is how to store the path in the .ini file. My suggestion would be to write it in JSON format, then pass it to json.loads(), which will get you the path like in my example.

base_path = '["results", 0, "locations"]'

You get the string '["results", 0, "locations"]' from your config (it might be a problem, is it?), pass it to json.loads

path = json.loads(path_string)

and you can use it in get_item function.

edited Mar 25, 2016 at 8:59

answered Mar 24, 2016 at 22:51

Ilya Peterov

2,0651 gold badge17 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jim Over a year ago

I'm going to test your suggestions, Ilya. I didn't think of loading values out of my own file using json so I rewrote it for configparser after hours of frustration.

Ilya Peterov Over a year ago

@jim I'm sorry that it caused you frustration, I suggested it because I thought you could put a JSON string (and any other string) in your config easily. Did my suggestion work after you rewrote your program to use configparser?

jim Over a year ago

sorry for the misunderstanding, I did not mean that you caused me frustration at all - it was all self-inflicted :) . Your answer is great and I'm using it test results from each web service provider. You really made my day with your suggestion and I appreciate it. I'll mark this as the answer.

Ilya Peterov Over a year ago

Very nice to hear that:) Really glad that I could be of any help!

Collectives™ on Stack Overflow

How to Create a Dynamic Path for Grabbing Variables from JSON in Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related