Compare list of datetimes with datetime in Python

Question

I have a list of datetime objects and would like to find the ones which are within a certain time frame:

import datetime

dates = [ datetime.datetime(2007, 1, 2, 0, 1),
          datetime.datetime(2007, 1, 3, 0, 2),
          datetime.datetime(2007, 1, 4, 0, 3),
          datetime.datetime(2007, 1, 5, 0, 4),
          datetime.datetime(2007, 1, 6, 0, 5),
          datetime.datetime(2007, 1, 7, 0, 6) ]
#in reality this is a list of over 25000 dates

mask = (dates>datetime.datetime(2007,1,3)) & \
       (dates<datetime.datetime(2007,1,6))

However, this results in the following error: "TypeError: can't compare datetime.datetime to list"

How can I fix my code?

possible duplicate of python: deleting list elements based on condition — Don Kirkby
– Don Kirkby, Commented Jan 3, 2013 at 21:00

Martijn Pieters · Accepted Answer · 2013-01-03 16:53:20Z

12

If your dates list is in sorted order, you can use the bisect module:

>>> import bisect
>>> bisect.bisect_right(dates, datetime.datetime(2007,1,3))
1
>>> bisect.bisect_left(dates, datetime.datetime(2007,1,6))
4

The .bisect_* functions return indices into the dates list:

>>> lower = bisect.bisect_right(dates, datetime.datetime(2007,1,3))
>>> upper = bisect.bisect_left(dates, datetime.datetime(2007,1,6))
>>> mask = dates[lower:upper]
>>> mask
[datetime.datetime(2007, 1, 3, 0, 2), datetime.datetime(2007, 1, 4, 0, 3), datetime.datetime(2007, 1, 5, 0, 4)]

edited Jan 3, 2013 at 16:53

answered Jan 3, 2013 at 16:39

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Zeugma Over a year ago

I would disagree with you: doesn't bisect_left include the element in argument if it is present in the list ?

Martijn Pieters Over a year ago

@Boud: That's the point of using bisect, it's to find the range of datetime objects that fall between the two values. Or do you mean it inserts it into the list? No, it doesn't do that, it simply returns the insertion point; use insort_*() to do that.

Zeugma Over a year ago

OP is asking for strict inequality

Martijn Pieters Over a year ago

@Boud: it's easy enough to adjust from _left to _right in that case. I've done so in my update.

HyperCube Over a year ago

You were right, it is the fastest method (2.48 us per loop)! Thank you!

|

Andy Hayden · Accepted Answer · 2013-01-03 16:48:45Z

12

You can mask a numpy.array in the syntax you describe (but not a list):

import numpy as np

date1 = np.array(dates)
mask = (dates1 > datetime.datetime(2007,1,3)) & \
       (dates1 < datetime.datetime(2007,1,6))

In [14]: mask
Out[14]: array([False,  True,  True,  True, False, False], dtype=bool)

In [15]: dates1[mask]
Out[15]: array([2007-01-03 00:02:00, 2007-01-04 00:03:00, 2007-01-05 00:04:00], dtype=object)

(since this question is tagged numpy, presumably this is what you were intending.)

edited Jan 3, 2013 at 16:48

answered Jan 3, 2013 at 16:39

Andy Hayden

378k110 gold badges640 silver badges546 bronze badges

3 Comments

HyperCube Over a year ago

The conversion np.array(dates) of my list of datetime objects to the numpy array takes almost 1 second....is there a faster conversion, since I have to do this repetitively?

Andy Hayden Over a year ago

don't start with a list, what do you have before that? 1s sounds long. I think worth looking into pandas for efficient datetime arrays. (pd.to_datetime(dates))

AskNature Over a year ago

I wonder if there is two period time I need to find in a array. How do I merge the mask?

Inbar Rose · Accepted Answer · 2013-01-03 16:40:23Z

5

import datetime

dates = [ datetime.datetime(2007, 1, 2, 0, 1),
          datetime.datetime(2007, 1, 3, 0, 2),
          datetime.datetime(2007, 1, 4, 0, 3),
          datetime.datetime(2007, 1, 5, 0, 4),
          datetime.datetime(2007, 1, 6, 0, 5),
          datetime.datetime(2007, 1, 7, 0, 6) ]


within = [date for date in dates if datetime.datetime(2007,1,3) < date < datetime.datetime(2007,1,6)]

yields:

[datetime.datetime(2007, 1, 3, 0, 2), 
 datetime.datetime(2007, 1, 4, 0, 3), 
 datetime.datetime(2007, 1, 5, 0, 4)]

answered Jan 3, 2013 at 16:40

Inbar Rose

43.7k24 gold badges91 silver badges137 bronze badges

7 Comments

HyperCube Over a year ago

This is even faster (7.82 us per loop), than turning it into a numpy array (approach from hayden) (122 us per loop)

Martijn Pieters Over a year ago

@HyperCube: I bet using bisect on a sorted list is even faster though. bisect does not traverse the whole list, it's methods bounded by O(log n) complexity, while this answer requires you to access the whole list (O(n) complexity).

HyperCube Over a year ago

However, I need the position in the list. @MartijnPieters: I will try this out next ;)

Inbar Rose Over a year ago

@MartijnPieters - also bisect requires a sorted list.

Martijn Pieters Over a year ago

@InbarRose: indeed, as I stated in my answer. :-)

|

Collectives™ on Stack Overflow

Compare list of datetimes with datetime in Python

3 Answers 3

7 Comments

3 Comments

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

3 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related