Merge multiple python dictionaries into dynamic file

Question

I want to create a single file combining multiple python dictionaries about costal weather conditions (tides, wind, etc) that can be updated day to day. The data comes from multiple APIs and websites, each of which I convert to a python dictionary, and merge into a single dictionary using the following line of code:

OneDayWeather_data = {'Willydata' : Willydata, 'Bureau of Meteorology' : BoMdata, 'WeatherZone' : WZdata}

My goal is to sample the sites each day; and update a single file with each day's weather and forecast across the sites. I am thinking the best way to do this is to create a new top level to the hierarchy using the date. So it would be like something like:

Weather_data['18/07/2017']['Willy']['Winds']

Weather_data['18/07/2017']['BoMdata']['Winds']

For each day, I would then add a new top level entry for the new day's data, i.e.

AllWeatherData['19/07/2017']['Willy']['Winds']

I have tried this using a variety of methods suggested from stack overflow (Full disclosure: I'm pretty new to Python). For example,

# write the initial file
with open('test.json', 'w') as f:
    json.dump(OneDayWeather_data, f)    

# open the initial file and attempt to append
with open('test.json','r+') as f:
    dic = dict(json.load(f))
    dic.update(OneDayWeather_data)
    json.dump(dic, f)

# reopen the appended file
with open('test.json', 'r') as f2:
    json_object = json.load(f2)

...but I keep getting errors (in this case: ValueError(errmsg("Extra data", s, end, len(s))) when I try to reopen). Hoping someone w some expertise can weigh in on how to approach this problem.

Thanks!

What about using TinyDB? tinydb.readthedocs.io/en/latest

Grimmy
– Grimmy

2017-07-18 14:20:27 +00:00
Commented Jul 18, 2017 at 14:20 — Grimmy
– Grimmy, Commented Jul 18, 2017 at 14:20

Sanghack Lee · Accepted Answer · 2017-07-18 14:19:08Z

0

You are actually appending the update dict to the existing dict

# write the initial file
import json

OneDayWeather_data = {'a':'b'}

with open('test.json', 'w') as f:
    json.dump(OneDayWeather_data, f)

OneDayWeather_data = {'c':'d'}

# open the initial file and attempt to append
with open('test.json','r+') as f:
    dic = dict(json.load(f))
    dic.update(OneDayWeather_data)
    json.dump(dic, f)

# reopen the appended file
with open('test.json', 'r') as f2:
    json_object = json.load(f2)

At this stage, your test.json looks like

{"a": "b"}{"a": "b", "c": "d"}

You may separate read/update/write

with open('test.json','r') as f:
    dic = dict(json.load(f))
    dic.update(OneDayWeather_data)
with open('test.json', 'w') as f:
    json.dump(dic, f)

Similar answer can be found in How to append in a json file in Python?

answered Jul 18, 2017 at 14:19

Sanghack Lee

1012 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thomas Carlson Over a year ago

Thanks! Worked perfectly.

Collectives™ on Stack Overflow

Merge multiple python dictionaries into dynamic file

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related