Fail to transfer the list in data frame to numpy array with python-pandas

Question

For a data frame df:

name       list1                    list2
a          [1, 3, 10, 12, 20..]     [2, 6, 23, 29...]
b          [2, 10, 14, 3]           [4, 7, 8, 13...]
c          []                       [98, 101, 200]
...

I want to transfer the list1 and list2 to np.array and then hstack them. Here is what I did:

df.pv = df.apply(lambda row: np.hstack((np.asarray(row.list1), np.asarray(row.list2))), axis=1)

And I got such an error:

ValueError: Shape of passed values is (138493, 175), indices imply (138493, 4)

Where 138493==len(df)

Please note that some value in list1 and list2 is empty list, []. And the length of list are different among rows. Do you know what is the reason how can I fix the problem? Thanks in advance!

EDIT:

When I just try to convert one list to array:

df.apply(lambda row: np.asarray(row.list1), axis=1)

An error also occurs:

ValueError: Empty data passed with indices specified.

@ColonelBeauvel Thanks for your reply! Isn't the sample above reproducible? — user5779223
– user5779223, Commented Oct 6, 2016 at 10:26
@user5779223 How did you create you dataframe, that's what he meant — MMF
– MMF, Commented Oct 6, 2016 at 10:32
@MMF I read in a data set and convert it to the form like this. Indeed I still don't know what information you need? — user5779223
– user5779223, Commented Oct 6, 2016 at 10:33

Shivam Gaur · Accepted Answer · 2016-10-11 06:00:30Z

1

Your apply function is almost correct. All you have to do - convert the output of the np.hstack() function back to a python list.

df.apply(lambda row: list(np.hstack((np.asarray(row.list1), np.asarray(row.list2)))), axis=1)

The code is shown below (including the df creation):

df = pd.DataFrame([('a',[1, 3, 10, 12, 20],[2, 6, 23, 29]),
                   ('b',[2, 10, 1.4, 3],[4, 7, 8, 13]),
                   ('c',[],[98, 101, 200])],
                   columns = ['name','list1','list2'])

df['list3'] = df.apply(lambda row: list(np.hstack((np.asarray(row.list1), np.asarray(row.list2)))), axis=1)

print(df)

Output:

0              [1, 3, 10, 12, 20, 2, 6, 23, 29]
1    [2.0, 10.0, 1.4, 3.0, 4.0, 7.0, 8.0, 13.0]
2                          [98.0, 101.0, 200.0]
Name: list3, dtype: object

If you want a numpy array, the only way I could get it to work is:

df['list3'] = df['list3'].apply(lambda x: np.array(x))

print(type(df['list3'].ix[0]))
Out[] : numpy.ndarray

edited Oct 11, 2016 at 6:00

answered Oct 6, 2016 at 10:54

Shivam Gaur

1,06210 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user5779223 Over a year ago

Thanks for your answer, but what if I wanted it be a numpy array?

Collectives™ on Stack Overflow

Fail to transfer the list in data frame to numpy array with python-pandas

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related