How to index a complex nested JSON in Redis for querying

Question

I need some help with understanding how i can store and index all the elements of the below JSON in REDIS. My Goal is to store all the AWS DESCRIBE_INSTACE CLI Data in REDIS for all EC2 instances and query the Indexed Data to return all the EC2 Instances with a Particular Matching String.

For EG - I need to be able to find and return all the EC2 Instances who belong to a particular SUBNET/VPC

Can this be done in REDIS?

"Reservations": [
        {
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "PublicDnsName": "",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "EbsOptimized": "false",
                    "LaunchTime": "xxxxxxxxx",
                    "PrivateIpAddress": "x.x.x.x",
                    "ProductCodes": [],
                    "VpcId": "xxxxx",
                    "StateTransitionReason": "",
                    "InstanceId": "i-xxxxxxx",
                    "EnaSupport": "true",
                    "ImageId": "ami-xxxxx",
                    "PrivateDnsName": "ip-xxxxxx.ec2.internal",
                    "KeyName": "xxxxxxv",
                    "SecurityGroups": [
                        {
                            "GroupName": "xxx",
                            "GroupId": "sg-xxxx"
                        },
                        {
                            "GroupName": "xxxxxx",
                            "GroupId": "sg-xxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        },
                        {
                            "GroupName": "xxxxx",
                            "GroupId": "sg-xxxxxx"
                        }
                    ],
                    "ClientToken": "xxxxx",
                    "SubnetId": "subnet-xxxxx",
                    "InstanceType": "t2.micro",
                    "NetworkInterfaces": [
                        {
                            "Status": "in-use",
                            "MacAddress": "xxxxxxxx",
                            "SourceDestCheck": "true",
                            "VpcId": "vpc-xxxxx",
                            "Description": "",
                            "NetworkInterfaceId": "eni-xxxxx",
                            "PrivateIpAddresses": [
                                {
                                    "PrivateDnsName": "ip-xx-ec2.internal",
                                    "Primary": "true",
                                    "PrivateIpAddress": "xxxxx"
                                }
                            ],
                            "PrivateDnsName": "ip-xxxx-xx.ec2.internal",
                            "Attachment": {
                                "Status": "attached",
                                "DeviceIndex": 0,
                                "DeleteOnTermination": "true",
                                "AttachmentId": "eni-attach-xxxxx",
                                "AttachTime": "2017-0xxxxx"
                            },
                            "Groups": [
                                {
                                    "GroupName": "xx",
                                    "GroupId": "sg-xxxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxx"
                                },
                                {
                                    "GroupName": "xxxx",
                                    "GroupId": "sg-xxxx"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "OwnerId": "xxx",
                            "SubnetId": "subnet-xxxx",
                            "PrivateIpAddress": "1xxxx"
                        }
                    ],
                    "SourceDestCheck": "true",
                    "Placement": {
                        "Tenancy": "default",
                        "GroupName": "",
                        "AvailabilityZone": "us-xxxxxxx"
                    },
                    "Hypervisor": "xen",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xxxxxx",
                            "Ebs": {
                                "Status": "attached",
                                "DeleteOnTermination": "true",
                                "VolumeId": "vol-xxxxxx",
                                "AttachTime": "2017-xxxxxxx"
                            }
                        }
                    ],
                    "Architecture": "x86_64",
                    "RootDeviceType": "ebs",
                    "IamInstanceProfile": {
                        "Id": "xxxxxxxx",
                        "Arn": "arn:aws:iam::xxxxxxx"
                    },
                    "RootDeviceName": "/dev/xxxxx",
                    "VirtualizationType": "hvm",
                    "Tags": [
                        {
                            "Value": "xxxxxx",
                            "Key": "aws:cloudformation:stack-name"
                        },
                        {
                            "Value": "xxxxxxx",
                            "Key": "aws:cloudformation:logical-id"
                        },
                        {
                            "Value": "arn:aws:cloudformation:xxxxxx",
                            "Key": "aws:cloudformation:stack-id"
                        }
                    ],
                    "AmiLaunchIndex": 0
                }
            ],
            "ReservationId": "r-xxxxx",
            "RequesterId": "xxxxx",
            "Groups": [],
            "OwnerId": "xxxxxx"
        }
    ]  
}

We have a module for indexing in redis, but it's not integrated with ReJSON yet. You can manually bind them together if you want, other people have. see redisearch.io — Not_a_Golfer
– Not_a_Golfer, Commented Nov 2, 2017 at 21:23

Alexander Gugel · Accepted Answer · 2021-01-24 16:13:35Z

Yes, this can be done. It's important to realize however that Redis itself doesn't really care about JSON, as such you'll have to massage your data into a format that can be queried efficiently. This "normalization" (let's call it that for now) will have to happen on an application level. In other words - you'll have to deal with this, not Redis.

For EG - I need to be able to find and return all the EC2 Instances who belong to a particular SUBNET/VPC

Let's take a look at this example. One way of modeling this data might be as follows:

Assign an unique id for each instance.
Store each instance as a JSON encoded blob under instance:{instance_id}.
Store instance ids under their respective instance_by_subnet:subnet-xxxxx set.

More concretely - using the example you provided - this might look as follows:

SET instance:1 {"Monitoring":.......} # Store first instance
SADD instance_by_subnet:subnet-abc123 1 # Add instance id to relevant set

This way you'd be able to determine with instances belong to a specific subset by calling SMEMBERS instance_by_subnet:subnet-abc123. You'd have to issue another GET command to actually read the JSON blobs.

There are multiple ways of maintaining those indices and a number of other data structures that come in handy when it comes to modeling your data. I wrote an article that covers this in a bit more detail in case you're interested.

Collectives™ on Stack Overflow

How to index a complex nested JSON in Redis for querying

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related