Serializing object references inside objects in python json

Question

I am attempting to saving my QGraphicsScene, where the items being drawn may contain references to other items on the scene.

This raises a major concern as when saving to json, and loading from it, the object may appear at correct position but the references is something I am unsure of how to implement.

Is there a way how I can store which object a reference is to? (the referred object is also stored in the json)

I use getstate and setstate methods to serialize my objects, and the ones that will never have references but are always referred are added to the json first so the reference objects are there when loaded.

Here is an MRE:

import json
class shape():
    def __init__(self, x, y, width, height):
        self.x = x
        self.y = y
        self.width = width
        self.height = height

    def __getstate__(self):
        return {
            "x": self.x,
            "y": self.y,
            "width": self.width,
            "height": self.height
        }

    def __setstate__(self, dict):
        pass

class line():
    def __init__(self, shape1=None, shape2=None):
        self.shape1 = shape1
        self.shape2 = shape2

    def __getstate__(self):
        return {
            "shape1": self.shape1,
            "shape2": self.shape2
        }

    def __setstate__(self, dict):
        self.shape1 = dict['shape1']
        self.shape2 = dict['shape2']

if __name__ == "__main__":
    shapes = [shape(4, 4 , 4, 4) for _ in range(4)]
    line1 = line(shapes[0], shapes[1])
    line2 = line(shapes[2], shapes[3])
    items = {
        "shapes": [item.__getstate__() for item in shapes],
        "lines": [line1.__getstate__(), line2.__getstate__()] 
    }
    with open("test.json", "w") as file:
        json.dump(items, file)
    with open("test.json", "r") as file:
        item = json.load(file)

From the above example, i want to retain the line1 and line2 to have references to the specific shapes even after loading.

You probably want to write a very simple example code, like a boiler-plate code just to show off your concept. Something that we can execute and try out to better understand what you need. See how to write a minimal working example for some guidelines. — Torxed
– Torxed, Commented Jun 8, 2020 at 14:03
Looks great. I have an answer in mind that MIGHT help you, but I'm not sure. — Torxed
– Torxed, Commented Jun 8, 2020 at 14:25

Torxed · Accepted Answer · 2020-06-08 14:37:02Z

0

You probably need a JSONEncoder of sorts. Something that can parse the input data to json.dump before it gets passed through the sanitizers of the json lib.

You can do this by hooking in a cls=Typer class like so:

from json import JSONEncoder, dumps, loads
from datetime import date, datetime

class JSON_Encoder:
    def _encode(obj):
        if isinstance(obj, dict):
            ## We'll need to iterate not just the value that default() usually gets passed
            ## But also iterate manually over each key: value pair in order to trap the keys.

            for key, val in list(obj.items()):
                if isinstance(val, dict):
                    val = loads(dumps(val, cls=JSON_Typer)) # This, is a EXTREMELY ugly hack..
                                                            # But it's the only quick way I can think of to 
                                                            # trigger a encoding of sub-dictionaries. (I'm also very tired, yolo!)
                else:
                    val = JSON_Encoder._encode(val)
                del(obj[key])
                obj[JSON_Encoder._encode(key)] = val
            return obj
        elif hasattr(obj, 'json'):
            return obj.json()
        elif isinstance(obj, (datetime, date)):
            return obj.isoformat()
        elif isinstance(obj, (list, set, tuple)):
            r = []
            for item in obj:
                r.append(loads(dumps(item, cls=JSON_Typer)))
            return r
        else:
            return obj

class JSON_Typer(JSONEncoder):
    def _encode(self, obj):
        return JSON_Encoder._encode(obj)

    def encode(self, obj):
        return super(JSON_Typer, self).encode(self._encode(obj))

This code is by far not perfect, I'm aware of many issues with it. But it gets the job done 99% of the time as long as you're careful with circular dependencies.

With the above code, all you need to do is two things. The first being implementing a json function as the JSON_Encoder will be looking for elif hasattr(obj, 'json'): (you could change this to to_json() or something if you feel that's more logical).

Secondly, when doing json.dump(items, file), simply change it to:

json.dump(items, file, cls=JSON_Typer)

As it will append the pre-parser for all the objects seen in the JSON structure. Note here that it will traverse the dictionary, so it might feel a bit slow if your JSON structure is massive. And you'll need to implement a logic to convert yout shape into a representation of it so you upon json.load can interpret the data and do a replacement of that position in the JSON.

This is the best way I've fond over the years, it's a bit hacky but I've had some success with it.

answered Jun 8, 2020 at 14:37

Torxed

23.6k15 gold badges91 silver badges135 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Blaine Over a year ago

I am still unsure how this solves my problem, since I can save the files just fine. How does this help me store references? since when i am loading the object i load the dictionary properties and create a new object with value from the dictionary. Which means the reference values have changed.

Torxed Over a year ago

@Blaine I might have missunderstood your problem then, since your MRE gave me: TypeError: Object of type shape is not JSON serializable and this code solves that issue.

Blaine Over a year ago

Oh well, anyways I fixed the issue by assigning an id to each object, i save the ones that are referred to first, assign an id, then use the same id to restore the reference. Thanks however, you taught me something new :)

Torxed Over a year ago

Glad you found a solution, please post it for future reference in case other people end up here and need the same solution and mark your answer as the correct one :)

Blaine Over a year ago

I just did that. I am unsure as to why you use JSON_Encoder, when a default method in JSON_Typer would have just done the same thing? I am quite unsure as to how this works however. My own extention to the JSONEncoder had a default(self, o) that returned o.__getstate__() since pickle uses getstate as well.

Blaine · Accepted Answer · 2020-06-09 07:34:42Z

While @Torxed answers serializability of python classes, My main objective was saving references to objects. But since at loading I was creating new objects, storing references in the json file became a challenge.

My approach was using hex(id(ref)) to assign id to the object, and the same ref id gets stored in the referring object.

shapeDict = {}
class shape():
    ...

    def __getstate__(self):
        return {
            "x": self.x,
            "y": self.y,
            "width": self.width,
            "height": self.height,
            "id": hex(id(self))
        }

    def __setstate__(self, dict):
        shapeDict[dict['id']] = self

class line():
    ...

    def __getstate__(self):
        return {
            "shape1": hex(id(self.shape1)),
            "shape2": hex(id(self.shape2))
        }

    def __setstate__(self, dict):
        self.shape1 = shapeDict[dict['shape1']]
        self.shape2 = shapeDict[dict['shape2']]

At load time I just create a dict of all shapes that I have added with their id as keys and the assign the new reference at load time.

Collectives™ on Stack Overflow

Serializing object references inside objects in python json

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related