Using Binary Values from AWS Console for DynamoDB

Question

We are using a set of Lambda functions written in Python to read and write data to a DynamoDB table, including a column which is compressed binary data. The Lambda functions work perfectly fine, able to decompress the binary data successfully and use it.

We recently received a question that required us to get into the console and investigate the binary data in the DynamoDB table. This is when we ran into issues.

When we compress the data, Python shows us something like this:

b'x\x9c\xed\x99\xcfO\xdb0\x14\xc7\xef\xfc\x15V\xcf\x14%M\xe9\xc6n]\t\x10\t\x95\n\x18\x97\xa9\xb2\x8c'

In the AWS Console, however, we see values that look more like this:

'eJzsveuOHUmSHvgqifo1i1U6/H7Zf1lkspoDMkkkydKMBIHoqa7daexEt7a6WoBD0LvL7GSeZJi5hceh5cnkYoaUutnD7zMPjzgR'

Putting the second value into Python and trying to decompress it doesn't work, even when I convert it to a bytes array or read it as a binary file or whatever.

Does anyone know a good way to get something closer to the first value through the AWS console, or at least convert the second value to the first value, so that we can run it through the decompress function and get the actual value? If not, can someone at least give me the terminology of what to call the format of these two? They are obviously both binary data, but I'm sure there's a good name to describe how they are encoded or represented here.

Thanks!

What DynamoDB attribute type is this attribute? I'm assuming type B (binary). How are you presenting the value to the PutItem call? — jarmod
– jarmod, Commented Apr 14, 2022 at 15:34
@jarmod The attribute is B (binary), as you assumed. I am taking the output from the compression (which comes out as a Python binary type), place it into a JSON document, and then call the boto3 put_item function call to save it to DynamoDB. — Dave McGinnis
– Dave McGinnis, Commented Apr 14, 2022 at 15:45
The AWS console view seems to be base64-encoded. If you decode it (base64.b64decode('eJzsv...')) you'll see that it comes back as b'x\x9c\xec\xbd...'. Interestingly that's not the same bytes you show above, not sure why right now. Not aware of any options to control how the AWS console displays this. — jarmod
– jarmod, Commented Apr 14, 2022 at 16:18
@jarmod That's exactly it. The two I put up there were two different records, which is why the bytes didn't exactly match. I tried running against it and got some valid output though. Please make an answer for this and I'll accept it. — Dave McGinnis
– Dave McGinnis, Commented Apr 14, 2022 at 16:27

jarmod · Accepted Answer · 2024-09-12 13:32:21Z

3

The DynamoDB console is showing you a base64-encoded representation of the underlying binary attribute value.

I'm assuming that's a reasonable thing to do so that the console can safely display arbitrary binary attribute values.

If you decode the displayed value, as follows, you can recover the original Python bytes:

import base64
rc = base64.b64decode('eJzsv...') # Result is: b'x\x9c\xec\xbd...'

edited Sep 12, 2024 at 13:32

answered Apr 14, 2022 at 16:35

jarmod

79.8k18 gold badges132 silver badges137 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using Binary Values from AWS Console for DynamoDB

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related