Python: Converting Entire Directory of JSON to Python Dictionaries to send to MongoDB

Question

I'm relatively new to Python, and extremely new to MongoDB (as such, I'll only be concerned with taking the text files and converting them). I'm currently trying to take a bunch of .txt files that are in JSON to move them into MongoDB. So, my approach is to open each file in the directory, read each line, convert it from JSON to a dictionary, and then over-write that line that was JSON as a dictionary. Then it'll be in a format to send to MongoDB

(If there's any flaw in my reasoning, please point it out)

At the moment, I've written this:

"""
Kalil's step by step iteration / write.

JSON dumps takes a python object and serializes it to JSON.
Loads takes a JSON string and turns it into a python dictionary.
So we return json.loads so that we can take that JSON string from the tweet and save it as a dictionary for Pymongo
"""

import os
import json
import pymongo

rootdir='~/Tweets'

def convert(line):
    line = file.readline()
    d = json.loads(lines)
    return d


for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        f=open(file, 'r')
        lines = f.readlines()
        f.close()
        f=open(file, 'w')
        for line in lines:
            newline = convert(line)
            f.write(newline)
        f.close()

But it isn't writing. Which... As a rule of thumb, if you're not getting the effect that you're wanting, you're making a mistake somewhere.

Does anyone have any suggestions?

Trevor · Accepted Answer · 2012-07-18 16:19:47Z

3

When you decode a json file you don't need to convert line by line as the parser will iterate over the file for you (that is unless you have one json document per line).

Once you've loaded the json document you'll have a dictionary which is a data structure and cannot be directly written back to file without first serializing it into a certain format such as json, yaml or many others (the format mongodb uses is called bson but your driver will handle the encoding for you).

The overall process to load a json file and dump it into mongo is actually pretty simple and looks something like this:

import json
from glob import glob
from pymongo import Connection

db = Connection().test

for filename in glob('~/Tweets/*.txt'):
    with open(filename) as fp:
        doc = json.load(fp)

    db.tweets.save(doc)

answered Jul 18, 2012 at 16:19

Trevor

9,6082 gold badges27 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Noc Over a year ago

I'm an idiot. Thank you so much. I can't believe I didn't realize that huge, gaping error in logic regarding the python dictionary as a piece of memory.

Hassek · Accepted Answer · 2012-07-18 16:14:16Z

1

a dictionary in python is an object that lives within the program, you can't save the dictionary directly to a file unless you pickle it (pickling is a way to save objects in files so you can retrieve it latter). Now I think a better approach would be to read the lines from the file, load the json which converts that json to a dictionary and save that info into mongodb right away, no need to save that info into a file.

answered Jul 18, 2012 at 16:14

Hassek

9,0256 gold badges51 silver badges59 bronze badges

Collectives™ on Stack Overflow

Python: Converting Entire Directory of JSON to Python Dictionaries to send to MongoDB

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related