Create a Record array from a list of dictionaries

Question

Given a list of dictionaries as follows:

dict_data = [
    {'name': 'r1', 'interval': [1800.0, 1900.0], 'bool_condition': [True, False]},
    {'name': 'r2', 'interval': [1600.0, 1500.0], 'bool_condition': [False]},
    {'name': 'r3', 'interval': [1400.0, 1600.0], 'bool_condition': [True]}
]

I would like to create a record array from the dictionaries data. But when I try the following I get ValueError

import numpy as np
dt = np.dtype([
('name', np.str_, 50), ('interval', np.float64, (2,)),
    ('bool_condition', np.bool)
])
values = [tuple(val.values()) for val in dict_data]
arr = np.rec.array(values, dtype=dt)

Error: ValueError: cannot set an array element with a sequence

I would like to know how could I have a more correct dtype and then create the record array from the list of dictionaries.

B. M. · Accepted Answer · 2016-05-10 09:37:13Z

4

It's very convenient to do that with pandas :

 In [247]: pd.DataFrame(dict_data)[['name','interval','bool_condition']].to_records(False)

Out[247]: 
rec.array([('r1', [1800.0, 1900.0], [True, False]),
 ('r2', [1600.0, 1500.0], [False]), ('r3', [1400.0, 1600.0], [True])], 
          dtype=[('name', 'O'), ('interval', 'O'), ('bool_condition', 'O')])

['name','interval','bool_condition'] ensure the order of the fields.

edited May 10, 2016 at 9:37

answered May 10, 2016 at 9:28

B. M.

18.7k2 gold badges40 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Eric Over a year ago

But this has thrown out all the dtypes!

B. M. Over a year ago

Yes. but since list and str are not aligned data, there is no performance issues here. so the dtype is IMHO not very important.

Eric Over a year ago

I think the fact that interval is aligned is sufficient reason

Glen Thompson Over a year ago

Is there a numpy implementation of this without using Pandas?

Eric · Accepted Answer · 2016-05-10 21:08:40Z

3

One problem is that the iteration of a dictionary does not preserve the order. You can see this by looking at print values[0] which gives ([1800.0, 1900.0], [True, False], 'r1') if I use your code.

Rather use

import numpy as np
dt = np.dtype([
    ('name', np.str_, 50),
    ('interval', np.float64, (2,)),
    ('bool_condition', np.bool)
])
values = [
    tuple([val['name'], val['interval'], val['bool_condition']])
    for val in dict_data
]
arr = np.rec.array(values, dtype=dt)

Another thing is that the bool_condition in your data is a list and not just a boolean. So you might want to change your dtype to:

dt = np.dtype([
    ('name', np.str_, 50),
    ('interval', np.float64, (2,)),
    ('bool_condition', list)
])

edited May 10, 2016 at 21:08

Eric

98.1k54 gold badges257 silver badges389 bronze badges

answered May 10, 2016 at 9:17

Fabian Rost

2,4232 gold badges17 silver badges27 bronze badges

1 Comment

Eric Over a year ago

Note that list is just translated into np.object_ here

Collectives™ on Stack Overflow

Create a Record array from a list of dictionaries

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related