Using Python to send HTTP request and parse JSON

Question

I am very new to Python and am trying to get a response back from the URL for every user in the user.csv file, return the JSON message, then parse it to a CSV. I'm not sure what to put in the return section or how to split out the message. This is a sample of the message that is returned to me when I get the HTTP request:

{"msg": {"mes": "four", "high": 1230, "low": 0}}"

Here is what I have so far but I am getting an error:

TypeError: expected string or buffer

import requests
import json
import csv

def getRows(data):
    return []

url = 'http://test_url'

with open('user.csv', 'rU') as data_file:
    data = csv.DictReader(data_file)
    for row in data:
        current_user = row['mdn']
        r = requests.get(url)
        data = json.loads(data)
        fname = "mydata.csv"
with open(fname,'wb') as outf:
    outcsv = csv.writer(outf)
    outcsv.writerows(getRows(data))

can you give more detail (or maybe an example) on the content of your input csv file (mdn.csv)? also, can you give more detail about what role does the "current_user" has in the request? If you just call r = requests.get(url) for every user, you will always get the same response — caspillaga
– caspillaga, Commented Feb 9, 2017 at 1:29
the input csv file has a list of users. each one is going to hit the URL and return a JSON message similar to the one I provided as a sample. I then want to take the user, message, and parse it and convert the entire list with the JSON message to a csv. — c3win90
– c3win90, Commented Feb 9, 2017 at 1:32
Ok, but what's the difference between the request for user1 and user2? in your code, the request is identical for every user, so the responses for all of them will always be the same. Maybe you should be sending current_user in the request somehow? — caspillaga
– caspillaga, Commented Feb 9, 2017 at 1:38

caspillaga · Accepted Answer · 2017-02-09 02:07:39Z

1

With the information you provide, you can do something like this:

import requests
import csv

url = 'http://test_url'
with open('user.csv', 'rU') as data_file:
     data = csv.DictReader(data_file)
     for row in data:
         current_user = row['mdn']
         r = requests.get(url) # probably you also need to send current_user somehow. "r = requests.get(url+'?user='+current_user)" maybe?
         string_json = r.text.encode('utf-8')
         json = eval(string_json)   # only use this command if you are sure that string_json only contains a valid json and no malicious code instead
         with open('outputfile.csv','a') as outfile:
             outfile.write(current_user+';'+json["msg"]["mes"]+';'+json["msg"]["high"]+';'+json["msg"]["low"]+'\n')

Please provide more information about the HTTP request usage and about the format of your csv for a more precise answer

edited Feb 9, 2017 at 2:07

answered Feb 9, 2017 at 1:48

caspillaga

5634 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

c3win90 Over a year ago

Thanks! The HTTP request will give back information based on each user in the user.csv file, so the response will be different. In that case, you are right about needing to incorporate current_user. When running what you provided, I get TypeError: cannot concatenate 'str' and 'int' objects.

Collectives™ on Stack Overflow

Using Python to send HTTP request and parse JSON

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related