Python convert byte string to list of dictionaries based on newline

Question

I have a string byte

data = b'ra1,rb1,rc1\nra2,rb2,rc2\nra3,rb3,rc3\nra4,rb4,rc4' # No \n at the end

The result should be like

result = [
{"field1": "ra1", "field2": "rb1", "field3": "rc1"},
{"field1": "ra2", "field2": "rb2", "field3": "rc2"},
{"field1": "ra3", "field2": "rb3", "field3": "rc3"},
{"field1": "ra4", "field2": "rb4", "field3": "rc4"}
]

I tried

result = csv.DictReader(data, fieldnames=('field1', 'field2', 'field3'))

Please note that I'm not dealing with any CSV files here.
I know we need to pass file object as the first parameter to the above DictReader method to get the above result.
Is there any similar in-built method in python 3.6 to achieve my result?
Here I don't want to use loop since my data will be very huge.

Have a look at this answer, using the csv package's reader for a similar problem: stackoverflow.com/questions/53278358/… — sekky
– sekky, Commented May 27, 2019 at 17:48
The size of your data is irrelevant to whether a loop should be used or not. — Scott Hunter
– Scott Hunter, Commented May 27, 2019 at 17:49
@sekky, Even I don't want to use pandas package since it is huge in package size and also having lot of depending packages. For this single line, I don't want to waste 27 MB in my application server. — CMY
– CMY, Commented May 27, 2019 at 17:57
@mkrieger1, I'm not handling any CSV files. This is the problem. This DictReader method is available in CSV package. — CMY
– CMY, Commented May 27, 2019 at 18:04

Devesh Kumar Singh · Accepted Answer · 2019-05-27 17:58:19Z

0

You can create a list of lists of all the values by splitting on \n and then ,

Then you can zip the keys with each value sublist to create your list of dictionaries

Hence the code will be

data = b'ra1,rb1,rc1\nra2,rb2,rc2\nra3,rb3,rc3\nra4,rb4,rc4'
keys = ['field1', 'field2', 'field3']

#Create list of lists for values
values  = [ item.split(',') for item in data.decode('utf-8').splitlines() ]
#[['ra1', 'rb1', 'rc1'], ['ra2', 'rb2', 'rc2'], ['ra3', 'rb3', 'rc3'], ['ra4', 'rb4', 'rc4']]

#Create result dictionary by zipping keys and values
result = [dict(zip(keys, value)) for value in values]
print(result)

And the output will be

[{'field1': 'ra1', 'field2': 'rb1', 'field3': 'rc1'}, 
{'field1': 'ra2', 'field2': 'rb2', 'field3': 'rc2'}, 
{'field1': 'ra3', 'field2': 'rb3', 'field3': 'rc3'}, 
{'field1': 'ra4', 'field2': 'rb4', 'field3': 'rc4'}]

answered May 27, 2019 at 17:58

Devesh Kumar Singh

20.5k5 gold badges25 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

CMY Over a year ago

Is there any python in-built method to achieve this without these many loops? Since my data is very large, this will take twice the time.

Devesh Kumar Singh Over a year ago

Well you need to go through all items at0least once to process them, which the for loop does (it covers all items in one go, you can see the output of values list for the same), so you cannot avoid it @CMY All other solutions need to do it as well

Devesh Kumar Singh Over a year ago

Hi @CMY If you have any more questions let me know :) Otherwise if the answer helped you , please consider marking it as accepted by clicking the tick next to the answer :) I would also suggest you to take a look at stackoverflow.com/help/someone-answers

Onyambu · Accepted Answer · 2019-05-27 19:08:21Z

0

import pandas as pd, io
data = b'ra1,rb1,rc1\nra2,rb2,rc2\nra3,rb3,rc3\nra4,rb4,rc4'
names = {0:'fields1',1:'fields2',2:'fields3'}

pd.read_csv(io.StringIO(data.decode('utf8')),header=None).rename(names,axis=1).to_dict('records')

[{'fields1': 'ra1', 'fields2': 'rb1', 'fields3': 'rc1'},
 {'fields1': 'ra2', 'fields2': 'rb2', 'fields3': 'rc2'},
 {'fields1': 'ra3', 'fields2': 'rb3', 'fields3': 'rc3'},
 {'fields1': 'ra4', 'fields2': 'rb4', 'fields3': 'rc4'}]

answered May 27, 2019 at 19:08

Onyambu

80.3k3 gold badges29 silver badges65 bronze badges

Collectives™ on Stack Overflow

Python convert byte string to list of dictionaries based on newline

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related