How to store a Complex Nested JSON in Redis using Python

Question

As i am new to redis, i need some guidance on how we can store the below complex json in REDIS so that we can access the Elements of the JSON from REDIS -

"Reservations": [
        {
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "PublicDnsName": "",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "EbsOptimized": "false",
                    "LaunchTime": "xxxxxxxxx",
                    "PrivateIpAddress": "x.x.x.x",
                    "ProductCodes": [],
                    "VpcId": "xxxxx",
                    "StateTransitionReason": "",
                    "InstanceId": "i-xxxxxxx",
                    "EnaSupport": "true",
                    "ImageId": "ami-xxxxx",
                    "PrivateDnsName": "ip-xxxxxx.ec2.internal",
                    "KeyName": "xxxxxxv",
                    "SecurityGroups": [
                        {
                            "GroupName": "xxx",
                            "GroupId": "sg-xxxx"
                        },
                        {
                            "GroupName": "xxxxxx",
                            "GroupId": "sg-xxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        }
                    ],
                    "ClientToken": "xxxxx",
                    "SubnetId": "subnet-xxxxx",
                    "InstanceType": "t2.micro",
                    "NetworkInterfaces": [
                        {
                            "Status": "in-use",
                            "MacAddress": "xxxxxxxx",
                            "SourceDestCheck": "true",
                            "VpcId": "vpc-xxxxx",
                            "Description": "",
                            "NetworkInterfaceId": "eni-xxxxx",
                            "PrivateIpAddresses": [
                                {
                                    "PrivateDnsName": "ip-xx-ec2.internal",
                                    "Primary": "true",
                                    "PrivateIpAddress": "xxxxx"
                                }
                            ],
                            "PrivateDnsName": "ip-xxxx-xx.ec2.internal",
                            "Attachment": {
                                "Status": "attached",
                                "DeviceIndex": 0,
                                "DeleteOnTermination": "true",
                                "AttachmentId": "eni-attach-xxxxx",
                                "AttachTime": "2017-0xxxxx"
                            },
                            "Groups": [
                                {
                                    "GroupName": "xx",
                                    "GroupId": "sg-xxxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxxx"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "OwnerId": "xxx",
                            "SubnetId": "subnet-xxxx",
                            "PrivateIpAddress": "1xxxx"
                        }
                    ],
                    "SourceDestCheck": "true",
                    "Placement": {
                        "Tenancy": "default",
                        "GroupName": "",
                        "AvailabilityZone": "us-xxxxxxx"
                    },
                    "Hypervisor": "xen",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xxxxxx",
                            "Ebs": {
                                "Status": "attached",
                                "DeleteOnTermination": "true",
                                "VolumeId": "vol-xxxxxx",
                                "AttachTime": "2017-xxxxxxx"
                            }
                        }
                    ],
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "IamInstanceProfile": {
                        "Id": "xxxxxxxx",
                        "Arn": "arn:aws:iam::xxxxxxx"
                    },
                    "RootDeviceName": "/dev/xxxxx",
                    "VirtualizationType": "hvm",
                    "Tags": [
                        {
                            "Value": "xxxxxx",
                            "Key": "aws:cloudformation:stack-name"
                        },
                        {
                            "Value": "xxxxxxx",
                            "Key": "aws:cloudformation:logical-id"
                        },
                        {
                            "Value": "arn:aws:cloudformation:xxxxxx",
                            "Key": "aws:cloudformation:stack-id"
                        }
                    ],
                    "AmiLaunchIndex": 0
                }
            ],
            "ReservationId": "r-xxxxx",
            "RequesterId": "xxxxx",
            "Groups": [],
            "OwnerId": "xxxxxx"
        }
    ] 
}

I need to store this in such a way that i query the IP/Hostname/InstanceID to get all the elements that are present in the JSON.

I need some guidance on the above.

Guy Korland · Accepted Answer · 2021-02-13 20:16:00Z

10

You can't directly do that, but luckily there is a new Redis module called RedisJSON that does exactly what you need, and it has a nice Python binding as well. You can launch a RedisJSON docker container or use Redis 4.0+, then download/compile and install RedisJSON and configure Redis to load it, and it adds native commands for JSON manipulation.

It lets you store JSON documents in Redis, and then either fetch or modify a specific element in the document tree, without retrieving (or internally even parsing) the document. Its Python client even lets you store python dicts and converts them to JSON automatically.

ReJSON module: http://redisjon.io

Python client: https://pypi.python.org/pypi/rejson

Example:

from rejson import Client, Path

rj = Client(host='localhost', port=6379)

# Set the key `obj` to some object
obj = {
    'answer': 42,
    'arr': [None, True, 3.14],
    'truth': {
        'coord': 'out there'
    }
}
rj.jsonset('obj', Path.rootPath(), obj)

# Get something
print 'Is there anybody... {}?'.format(
    rj.jsonget('obj', Path('.truth.coord'))
)

edited Feb 13, 2021 at 20:16

Guy Korland

9,59814 gold badges65 silver badges113 bronze badges

answered Nov 2, 2017 at 15:35

Not_a_Golfer

49.5k15 gold badges130 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Itamar Haber Over a year ago

Took the words out of my mouth :)

Rohit Sarkar Over a year ago

Works beautifully, i have a question though, if i want to print the Tags Elements of the JSON how do i do that, i tried the below, but it gives me an error - print(r.jsonget('obj', Path('.Reservations.Instances.Tags')))

Not_a_Golfer Over a year ago

@RohitSarkar it's unrelated to ReJSON, I think your path is wrong. From looking at the document I can see that it should be .Reservations[0].Instances[0].Tags

Rohit Sarkar Over a year ago

@Not_a_Golfer - Thanks that worked, sorry i am a bit of a noob with Python and Redis

Not_a_Golfer Over a year ago

it's more of a JSON thing, unrelated to both python and redis.

|

Adarsh · Accepted Answer · 2017-11-03 03:56:26Z

2

You can use the pickle module incase you don't want to use reJson.

To set data in redis:

def store_dict_data_in_redis(redis_client, key, data, ex=0):
'''
store dict data in redis by pickle dumps
:param redis_client: redis client used to connect to obtain key value
:param key: key name
:param data: dict value
:param ex: expiry
:return: None
'''
    if ex > 0:
         redis_client.set(key, pickle.dumps(data), ex=ex)
    else:
         redis_client.set(key, pickle.dumps(data))

To obtain value from redis:

def get_dict_data_from_redis(redis_client, key):
'''
obtain dict data from redis
:param redis_client: redis client used to connect to obtain key value
:param key: key name
:return: dict data stored in redis
'''
    data = redis_client.get(key)
    if data:
       try:
          return pickle.loads(data)
       except:
          return eval(data.decode())
    return {}

answered Nov 3, 2017 at 3:56

Adarsh

3,6434 gold badges24 silver badges45 bronze badges

3 Comments

Not_a_Golfer Over a year ago

You can't modify a pickled object inside redis, and if you load and serialize not only is it slow, but your data will be inconsistent really quickly.

Adarsh Over a year ago

The goal here was to store data in redis and not modify it along the fly. i knew tat you can use reJson to do so.Just gave an alternative :)

Adarsh Over a year ago

reJson also needs minimum redis requirement of 4.0 or above. link : rejson.io

Hamed · Accepted Answer · 2022-02-06 12:25:56Z

I used this method to save data in JSON format in Radis and it worked on my project properly, ‍‍‍‍

Note That I have defined an expiration time for data storage‍ ‍‍‍‍

import redis
import json

rs = redis.Redis(host="127.0.0.1", port=6379, decode_responses=True)

def set_json(data, expire_time):
   for key in data.keys():
       if rs.set(key, str(data[key]), expire_time) is False:
           return False
       return True

def get_json(topic):
    value = rs.mget(topic)
    if value is not None:
        return json.loads(json.dumps(value[0]))
    return False


data = {
        "topic": 
        {
            "key0":"value",
            "key1":
            [
                {"key3":1}
            ]
        }
    }

set_json(data, 50)

topic = get_json("topic")
print(topic)

Output :

>> {'key0': 'value', 'key1': [{'key3': 1}]}

Collectives™ on Stack Overflow

How to store a Complex Nested JSON in Redis using Python

3 Answers 3

7 Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related