How to fetch JSON data from API, format / encode / write to a file?

Question

I need to fetch some data from a weather API, extract certain info and send it to std. output (in my case this is the console/terminal; I am playing around with python API scripting and do not yet have a web site/app do show fetched data).

Example Python code from the API provider (simple to understand and works):

import urllib2
import json

API_KEY='mysuperawesomekey'


f = urllib2.urlopen('http://api.wunderground.com/api/' + API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

json_string = f.read()
parsed_json = json.loads(json_string)
location = parsed_json['location']['city']
temp_f = parsed_json['current_observation']['temp_f']

print "Current temperature in %s is: %s" % (location, temp_f)

f.close()

Since I am on a free plan, I don't want to use up my API requests but rather fetch the JSON data, save it to a file and use in another script to practice. I found a couple of solutions here on StackOverflow but neither seem to work completely (nice formatting of the file):

Attempt 1. to save the fetched data, add. to the orig. code above:

import io
import codecs

with open('data.json', 'w') as outfile:
json.dump(json_string, outfile, indent=4, sort_keys=True, separators=(",", ':'))

Attempt 2:

with codecs.open('data.json', 'w', 'utf8') as jasonfile:
  jasonfile.write(json.dumps(parsed_json, sort_keys = True, ensure_ascii=False))

Both of my attempts work ("kind of") as I do get a .json file. But, upon inspecting it in my editor (Atom) I am seeing this (first few soft-break lines):

Output:

"\n{\n  \"response\": {\n  \"version\":\"0.1\",\n\"termsofService\":\"http://www.wunderground.com/weather/api/d/terms.html\",\n  \"features\": {\n  \"geolookup\": 1\n  ,\n  \"conditions\": 1\n  }\n\t}\n\t\t,\t\"location\": {\n\t\t\"type\":\"CITY\",\n\t\t\"country\":\"US\",\n\t\t\"country_iso3166\":\"US\",\n\t\t\"country_name\":\"USA\",\n\t\t\"state\":\"IA\",\n\t\t\"city\":\"Cedar Rapids\",\n\t\t\"tz_short\":\"CDT\",\n\t\t\"tz_long\":\"America/Chicago\",\n\t\t\"lat\":\"41.97171021\",\n\t\t\"lon\":\"-91.65871429\",\n\t\t\"zip\":\...

It is all on one line, with newlines and tabs showing. I have a couple of questions:

How can I check what format / kind of data-type API is returning? (FOr example is it just a raw string which needs to be JSON-iffied?)
Is my JSON not written in a human readable format because of encoding, delimiters fault?
Line json_string = f.read() from example code seems to "mimic" working with a "real" internal file object, is this correct?

Many thanks!

Your first question - check HTTP Headers, if it has application/json, it's trying to serve json :) — iScrE4m
– iScrE4m, Commented Sep 8, 2016 at 14:11
Bingo! It's serving a json file! :) SO, they fetch the file with urlopen into f, then read() this file, here it gets muddy... Should I save f to my filesystem od this json_string which is the result of reading the fetched f? Or even further down the line, maybe after parsed_json = json.loads()? — Alexander Starbuck
– Alexander Starbuck, Commented Sep 8, 2016 at 14:17
seems like you need to decode your json,stackoverflow.com/questions/12965203/… — Pavneet_Singh
– Pavneet_Singh, Commented Sep 8, 2016 at 14:18
If you are storing the json just to mock a server and practice json on it, don't worry about how it looks, only worry if you can load it. Json as a format escapes almost everything iirc — iScrE4m
– iScrE4m, Commented Sep 8, 2016 at 14:20
Cool. I CAN load it and all, but I'd like to re-format it nicely so I can use it as a human too :), inspect bits etc. Or, I need to be able to print it "nicely" to the console... — Alexander Starbuck
– Alexander Starbuck, Commented Sep 8, 2016 at 14:22

Alexander Starbuck · Accepted Answer · 2016-09-08 14:33:51Z

Seems that it was quite a simple solution. In my original code, I was saving a "non-parsed" variable to a file:

import urllib2
import json

API_KEY='key'

f = urllib2.urlopen('http://api.wunderground.com/api/' 
+ API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

# Saving the below variable into a .json file gives all the JSON data
# on a single line
json_string = f.read()

with open('data.json', 'w') as outfile:
    json.dump(json_string, outfile, indent=4, sort_keys=True, separators=(",", ':'))

which, of course, resulted in a file with JSON data on a single line. What I should have been doing is saving the parsed variable:

import urllib2
import json

API_KEY='key'

f = urllib2.urlopen('http://api.wunderground.com/api/' + API_KEY + '/geolookup/conditions/q/IA/Cedar_Rapids.json')

json_string = f.read()

# THIS one is correct!
parsed_json = json.loads(json_string)

with open('data.json', 'w') as outfile:
  json.dump(parsed_json, outfile, indent=4, sort_keys=True, separators=(",", ':'))

Collectives™ on Stack Overflow

How to fetch JSON data from API, format / encode / write to a file?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related