Python how to open multiple json files in a directory

Question

I am trying to open a directory in which there are multiple json files, to then make a data frame of the data in each of them, I try this:

    for file in os.listdir('Datasets/'):
        json_read = pd.read_json(file)

However, it gives me an error: ValueError: Expected object or value

As I inspect the type of the files, it says they are class str. When opening a single file in the directory with read_json it does work correctly as the file is recognized as json. I am not quite sure why the files are turned into strings nor how to solve it. Do you have any tips?

Thanks in advance!

Are there only json files in the folder? There might be hidden files that are not valid JSON. Maybe filter only for files that end with a .json extension — mousetail
– mousetail, Commented Jun 7, 2022 at 12:57
This will happen if a file you're trying to read as JSON is not valid JSON. Also, assuming there are no errors, then json_read will only ever have the data for the last observed JSON file — jackal
– jackal, Commented Jun 7, 2022 at 13:39

Noob-Programmer · Accepted Answer · 2022-06-07 13:09:28Z

3

import os
import pandas as pd
base_dir = '/path/to/dir'

#Get all files in the directory

data_list = []
for file in os.listdir(base_dir):

    #If file is a json, construct it's full path and open it, append all json data to list
    if file.endswith('json'):
        json_path = os.path.join(base_dir, file)
        json_data = pd.read_json(json_path, lines=True)
        data_list.append(json_data)

print(data_list)

edited Jun 7, 2022 at 13:09

answered Jun 7, 2022 at 12:59

Noob-Programmer

927 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Shalvex Novachrono · Accepted Answer · 2023-08-27 16:08:41Z

import os

import json
#heres some information about get list of file in a folder: <https://www.geeksforgeeks.org/python-list-files-in-a-directory/>
#heres some information about how to open a json file: <https://www.geeksforgeeks.org/read-json-file-using-python/>
path = "./data"
file_list = os.listdir(path) #opens the directory form path and gets name of file

for i in range(len(file_list)): #loop the list with index number
  
  current = open(path+"/"+file_list[i]) #opens the file in order by current index number
  data = json.load(current) #loads the data form file

  for k in data['01']: # 
      print(k)

output

main.json :
{'name': 'Nava', 'Surname': 'No need for that'}
data.json :
{'name': 'Nava', 'watchs': 'Anime'}

heres a link to run code online https://replit.com/@Nava10Y/opening-open-multiple-json-files-in-a-directory

jackal · Accepted Answer · 2022-06-07 13:46:23Z

0

You probably need to build a list of DataFrames. You may not be able to process every file in the given directory so try this:

import pandas as pd
from glob import glob
from os.path import join
BASEDIR = 'Datasets'
dataframes = []
for file in glob(join(BASEDIR, '*.json')):
    try:
        dataframes.append(pd.read_json(file))
    except ValueError:
        print(f'Unable to process {file}')
print(f'Successfully constructed {len(dataframes)} dataframes')

answered Jun 7, 2022 at 13:46

jackal

29.1k3 gold badges10 silver badges28 bronze badges

Collectives™ on Stack Overflow

Python how to open multiple json files in a directory

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related