How do I convert binary data to hexadecimal through Python?

Question

So I'm trying to get the binary data field in a database as a hexadecimal string. I don't know if that is exactly what I'm looking for but that is my hunch. There is a column called status in my dataframe that says [binary data] in PostgreSQL but when I execute the following command it looks like this:

df = pd.read_sql_query("""SELECT * FROM public."Vehtek_SLG" LIMIT 1000""",con=engine.connect())

How do I get the actual data in that column?

Please don't post images of code, data, or Tracebacks. Copy and paste it as text then format it as code (select it and type ctrl-k) ... Discourage screenshots of code and/or errors...Why not upload images of code on SO when asking a question? ... You should not post code as an image because:... — wwii
– wwii, Commented May 18, 2021 at 13:47

Ravaal · Accepted Answer · 2021-05-19 15:48:00Z

2

It looks like your DataFrame has for each row a list of individual bytes instead of the entire hexadecimal bytes string. The Series df["status"].map(b"".join) will have the concatenated bytes strings.

import random

import pandas as pd


# Simulating lists of 10 bytes for each row
df = pd.DataFrame({
    "status": [
        [bytes([random.randint(0, 255)]) for _ in range(10)]
        for _ in range(5)
    ]
})

s = df["status"].map(b"".join)

Both objects look like:

# df
                                              status
0  [b'\xb3', b'f', b';', b'P', b'\xcb', b'\x9b', ...
1  [b'\xd2', b'\xe8', b'.', b'b', b'g', b'|', b'\...
2  [b'\xa7', b'\xe1', b'z', b'-', b'W', b'\xb8', ...
3  [b'\xc5', b'\xa9', b'\xd5', b'\xde', b'\x1d', ...
4  [b'\xa3', b'b', b')', b'\xe3', b'5', b'`', b'\...

# s
0       b'\xb3f;P\xcb\x9bi\xb0\x9e\xfd'
1            b'\xd2\xe8.bg|\x94O\x90\n'
2       b'\xa7\xe1z-W\xb8\xc2\x84\xb91'
3    b'\xc5\xa9\xd5\xde\x1d\x02*}I\x15'
4                b'\xa3b)\xe35`\x0ed#g'
Name: status, dtype: object

After coverting the status field to binary we can then use the following to make it hexadecimal.

df['status'] = s.apply(bytes.hex)

And now here is your field!

df['status'].head()

0    1f8b0800000000000400c554cd8ed33010beafb4ef6045...
1    1f8b0800000000000400c554cd8ed33010beafb4ef6045...
2    1f8b0800000000000400c554cd6e9b4010be47ca3bac50...
3    1f8b0800000000000400c554cd6e9b4010be47ca3bac50...
4    1f8b0800000000000400c554cd6e9b4010be47ca3bac50...

edited May 19, 2021 at 15:48

Ravaal

3,3697 gold badges45 silver badges69 bronze badges

answered May 19, 2021 at 12:10

Roméo Després

2,2734 gold badges21 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ravaal Over a year ago

Ok. So is s a hexadecimal string now?

Roméo Després Over a year ago

It's a pandas Series containing Python bytes. If you want it to contain the hexadecimal representation of those bytes as str, you can use s.apply(bytes.hex).

Collectives™ on Stack Overflow

How do I convert binary data to hexadecimal through Python?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related