Creating dask dataframe from array doesn't keep column types

Question

I'm trying to create a dask dataframe from a numpy array. For that, I need to specify the column types. As suggested in dask documentation, I use for that a pandas empty dataframe. This doesn't throw an error, however all the data types are created as object. I need to use the empty Pandas dataframe, how to make this work?

import pandas as pd
import dask.dataframe as dd

array = np.array([(1.5, 2, 3, datetime(2000,1,1)), (4, 5, 6, datetime(2001, 2, 2))])
meta = pd.DataFrame({'col1': pd.Series(dtype='float64'),
                   'col2': pd.Series(dtype='float64'),
                   'col3': pd.Series(dtype='float64'),
                   'date1': pd.Series(dtype='datetime64[ns]')})
print(meta.dtypes)

>>> col1            float64
>>> col2            float64
>>> col3            float64
>>> date1    datetime64[ns]
>>> dtype: object

columns = ['col1', 'col2', 'col3', 'date1']
ddf = dd.from_array(array, columns=columns, meta=meta)
ddf.compute()

print(ddf.dtypes)

>>> col1     object
>>> col2     object
>>> col3     object
>>> date1    object
>>> dtype: object

How is this different from yesterday's question? stackoverflow.com/questions/70836962/… — hpaulj
– hpaulj, Commented Jan 25, 2022 at 12:52
It looks like this is a bug-- I would encourage you to submit an issue. Including @Alexandra Dudkina's additional solution could be helpful for debugging. — scj13
– scj13, Commented Jan 25, 2022 at 22:02
Update to my above comment-- this is not a bug, but more of a nuance around how the meta argument works. There is some discussion here on how to improve this. — scj13
– scj13, Commented Jan 27, 2022 at 17:39

Alexandra Dudkina · Accepted Answer · 2022-01-25 16:47:57Z

2

Could dtypes be set after dataframe creation?

import pandas as pd
import numpy as np
from datetime import datetime
import dask.dataframe as dd

array = np.array([(1.5, 2, 3, datetime(2000,1,1)), (4, 5, 6, datetime(2001, 2, 2))])

columns = ['col1', 'col2', 'col3', 'date1']
ddf = dd.from_array(array, columns = columns)
ddf.compute()

ddf = ddf.astype({'col1': 'float64','col2':'float64','col3':'float64','date1':'datetime64[ns]'})
print(ddf.dtypes)

answered Jan 25, 2022 at 16:47

Alexandra Dudkina

4,5123 gold badges18 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Green Noob · Accepted Answer · 2022-01-25 15:00:39Z

0

Does this work -

df = (pd.DataFrame(array, columns = ["col1", "col2", "col3", "col4"])
      .astype({"col1": "float64", 
               "col2": "float64", 
               "col3": "float64", 
               "col4": "datetime64[ns]"}))
ddf = dd.from_pandas(df, npartitions=10)

The output of ddf.dtypes gives me the correct data types.

answered Jan 25, 2022 at 15:00

Green Noob

4081 gold badge4 silver badges12 bronze badges

1 Comment

ps0604 Over a year ago

Thanks, but I need to create the data frame with dd.from_array

Collectives™ on Stack Overflow

Creating dask dataframe from array doesn't keep column types

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related