Pandas: Fixing datetime.time and datetime.datetime mix

Question

I have the following DataFrame, with 'Time' column with mixed datetime types:

time_series_slice = tmp_df['XXX']
time_series_slice['Time types'] = time_series_slice['Time'].apply(lambda row: type(row))
time_series_slice['Time types'].value_counts()

<class 'datetime.datetime'>    97367
<class 'datetime.time'>           25
Name: Time types, dtype: int64

I am having a problem converting this whole 'Time' column to Pandas datetime with pd.to_datetime() method due to:

TypeError: <class 'datetime.time'> is not convertible to datetime

Approach time_series_slice['Time'].apply(lambda x: pd.Timestamp(x)) also does not work:

TypeError: Cannot convert input [00:00:00] of type <class 'datetime.time'> to Timestamp

I figures that these 25 stupid rows with are giving me this headache, but I lack ideas on what to do with them.

Firstly, how do I force Pandas to display only these rows? time_series_slice[isinstance(time_series_slice['Time'], datetime.time)] gives me:

NameError: name 'datetime' is not defined

Secondly, how do I just convert all these values to Pandas datetime and move on? :(

UPDATE:

Adding sample data view:

0    2017-02-08 22:19:08.618000
1    2017-02-08 22:19:12.187000
2    2017-02-08 22:19:13.481000
3    2017-02-08 22:19:16.330000
4    2017-02-08 22:19:16.582000
Name: Time, dtype: object

UPDATE 2: Thanks to Wen-Ben's suggestion, I have filtered out the datetime.time rows, and they look as such:

time_series_slice['Time types'] = time_series_slice['Time'].apply(lambda row: type(row).__name__)
time_series_slice[time_series_slice['Time types'] == 'time']['Time']

96367    00:00:00
96368    00:00:00
96464    00:00:00
96465    00:00:00
96466    00:00:00
96467    00:00:00
96593    00:00:00
96862    00:00:00
Name: Time, dtype: object

Would the easiest way be to re-write them to a datetime.datetime object with all 0s?

Can you show us some of your data ?

BENY
– BENY

2019-04-17 01:27:15 +00:00
Commented Apr 17, 2019 at 1:27 — BENY
– BENY, Commented Apr 17, 2019 at 1:27
Done, see the update above.

Greem666
– Greem666

2019-04-17 01:29:15 +00:00
Commented Apr 17, 2019 at 1:29 — Greem666
– Greem666, Commented Apr 17, 2019 at 1:29

BENY · Accepted Answer · 2019-04-17 01:33:44Z

2

If you want to slice the those 5 rows

time_series_slice['Time types'] = time_series_slice['Time'].apply(lambda x : type(x).__name__)=='Timestamp'

time_series_slice['Time types'].value_counts()

time_series_slice[time_series_slice['Time types']=='datetime.time']

Then

We using to_datetime to convert

time_series_slice['Time']=pd.to_datetime(time_series_slice['Time'].astype(str))

edited Apr 17, 2019 at 1:33

answered Apr 17, 2019 at 1:29

BENY

324k22 gold badges176 silver badges250 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Greem666 Over a year ago

As I have mentioned in my original post, using pd.to_datetime() yields the error: TypeError: <class 'datetime.time'> is not convertible to datetime.

BENY Over a year ago

@Greem666 adding astype(str)

Greem666 Over a year ago

That hit the spot - everything got converted nicely. Thanks @Wen-Ben! Out of curiosity - how did just a simple string conversion actually fix the problem here? Is it a quirk of pd.to_datetime() implementation?

BENY Over a year ago

@Greem666 I feel like the problem is your columns already datetime.datetime and pandas prevent convert it, until it is string

Collectives™ on Stack Overflow

Pandas: Fixing datetime.time and datetime.datetime mix

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related