Reading tsv files with byte strings with pandas

Question

I have a tsv file with a column containing utf-8 encoded byte strings (e.g., b'La croisi\xc3\xa8re'). I am trying to read this file with the pandas method read_csv, but what I get is a column of strings, not byte strings (e.g., "b'La croisi\xc3\xa8re'").

How can I read that column as byte strings instead of regular strings in Python 3? I tried to use dtype={'my_bytestr_col': bytes} in read_csv with no luck.

Another way to put it: How can I go from something like "b'La croisi\xc3\xa8re'" to b'La croisi\xc3\xa8re'?

to go from something like "b'La croisi\xc3\xa8re'" to b'La croisi\xc3\xa8re' you can do data[2:-1].encode() — Equinox
– Equinox, Commented Mar 12, 2018 at 20:26
Not really, this returns the following wrongly encoded byte string: b'La croisi\xc3\x83\xc2\xa8re' — Oriol Nieto
– Oriol Nieto, Commented Mar 12, 2018 at 20:36

It_is_Chris · Accepted Answer · 2018-03-12 23:59:08Z

1

sample file:

    First Name  Last Name   bytes
0   foo          bar        b'La croisi\xc3\xa8re'

then try this:

import pandas as pd
import ast
df = pd.read_csv('file.tsv', sep='\t')
df['bytes'].apply(ast.literal_eval)

Out:

0    b'La croisi\xc3\xa8re'
Name: bytes, dtype: object

edited Mar 12, 2018 at 23:59

answered Mar 12, 2018 at 20:34

It_is_Chris

14.2k3 gold badges27 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Oriol Nieto Over a year ago

This doesn't work because it opens the utf-8 encoded byte string column from my tsv file into a simple string. Maybe there's a set of arguments I could pass to this function that would fix it?

It_is_Chris Over a year ago

bytes(str(df['my_bytestr_col']),'utf-8')

Oriol Nieto Over a year ago

This returns b"b'La croisi\\xc3\\xa8re'" in Python 3

It_is_Chris Over a year ago

OK, this should work for you: df['bytes'].apply(ast.literal_eval)

Collectives™ on Stack Overflow

Reading tsv files with byte strings with pandas

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related