Flattening JSON objects using Python

Question

I have a json object

sample_json ={
"workspaces": [
    {
        "wsid": "1",
        "wsname": "firstworkspace",
        "report":[{"reportname":"r1ws1"},{"reportname":"r2ws1"},{"reportname":"r1ws1"}]
    },
    {
        "wsid": "2",
        "wsname": "secondworkspace",
        "report":[{"reportname":"r1ws2"},{"reportname":"r2ws2"},{"reportname":"r1ws2"}]
    }
]}

I want to flatten/normalize the json so that it looks like this :

I can flatten each level using the pandas function "json_normalize" e.g Level 1

import pandas as pd
df1 = pd.json_normalize(sample_json, record_path=['workspaces'])
df1

e.g Level 2

import pandas as pd
df_api = pd.json_normalize(sample_json, record_path=['workspaces','report'])
df_api

I have read quiet a few articles such as this one but I am struggling to understand how to include information from the two levels (workspaces and reports) in a single dataframe. Any advice or help would be appreciated. Copy to sample data in a google colab notebook is here

this has nothing to do with JSON. There is no JSON involved, there is a dictioanry — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Nov 11, 2022 at 19:25

The Lazy Graybeard · Accepted Answer · 2022-11-11 21:47:03Z

The package jertl can be used for this.

In [1]: import jertl

In [2]: import pandas as pd

In [3]: sample_json = ... #removed for brevity

In [4]: pattern = '''
   ...: {
   ...: "workspaces": [*_,
   ...:     {
   ...:         "wsid":   wsid,
   ...:         "wsname": wsname,
   ...:         "report": [*_, {"reportname": reportname}, *_]
   ...:     },
   ...:     *_]
   ...: }
   ...: '''

In [5]: df = pd.DataFrame(match_.bindings for match_ in jertl.match_all(pattern, sample_json))

In [6]: df
Out[6]:
  wsid           wsname reportname
0    1   firstworkspace      r1ws1
1    1   firstworkspace      r2ws1
2    1   firstworkspace      r1ws1
3    2  secondworkspace      r1ws2
4    2  secondworkspace      r2ws2
5    2  secondworkspace      r1ws2

pattern defines the structure to look for. It is intentionally JSON like. wsid, wsname and reportname are variables. The *_s are splatted anonymous variables which match any number of list items.

Shameless self promotion alert. I am the author of jertl.

juanpa.arrivillaga · Accepted Answer · 2022-11-11 19:40:04Z

0

So, just normalize this yourself.

df = pd.DataFrame(
    [
        {"wsid": ws['wsid'], "wsname": ws['wsname'], **rep}
        for ws in sample_json["workspaces"] for rep in ws['report']
    ]
)

answered Nov 11, 2022 at 19:40

juanpa.arrivillaga

97.6k14 gold badges141 silver badges190 bronze badges

1 Comment

Steve Over a year ago

Thanks @juanpa.arrivillaga Can you explain the mechanics of what is going on here?

Collectives™ on Stack Overflow

Flattening JSON objects using Python

2 Answers 2

Shameless self promotion alert. I am the author of jertl.

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Shameless self promotion alert. I am the author of jertl.

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related