Pandas DataFrame How to query the closest datetime index?

Question

How do i query for the closest index from a Pandas DataFrame? The index is DatetimeIndex

2016-11-13 20:00:10.617989120   7.0 132.0
2016-11-13 22:00:00.022737152   1.0 128.0
2016-11-13 22:00:28.417561344   1.0 132.0

I tried this:

df.index.get_loc(df.index[0], method='nearest')

but it give me InvalidIndexError: Reindexing only valid with uniquely valued Index objects

Same error if I tried this:

dt = datetime.datetime.strptime("2016-11-13 22:01:25", "%Y-%m-%d %H:%M:%S")
df.index.get_loc(dt, method='nearest')

But if I remove method='nearest' it works, but that is not I want, I want to find the closest index from my query datetime

jezrael · Accepted Answer · 2017-02-16 06:28:26Z

77

It seems you need first get position by get_loc and then select by []:

dt = pd.to_datetime("2016-11-13 22:01:25.450")
print (dt)
2016-11-13 22:01:25.450000

print (df.index.get_loc(dt, method='nearest'))
2

idx = df.index[df.index.get_loc(dt, method='nearest')]
print (idx)
2016-11-13 22:00:28.417561344

#if need select row to Series use iloc
s = df.iloc[df.index.get_loc(dt, method='nearest')]
print (s)
b      1.0
c    132.0
Name: 2016-11-13 22:00:28.417561344, dtype: float64

edited Feb 16, 2017 at 6:28

answered Feb 16, 2017 at 6:20

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Bryan Fok Over a year ago

thank you for your solution. I believe your solution would work, but it just doesnt work on me... this is my index type <class 'pandas.tslib.Timestamp'> <class 'pandas.tseries.index.DatetimeIndex'>

jezrael Over a year ago

Can you explain more? Does it return wrong value? Problem is with this sample of with real data?

jezrael Over a year ago

Is possible remove duplicates in index ? idx = df.index.drop_duplicates().get_loc(dt, method='nearest')

fhchl Over a year ago

Is there any way to query for multiple date times simultaneously? get_loc only accepts scalar values.

Brueni92 Over a year ago

as mentioned by @HarryChil, DatetimeIndex.get_loc is now deprecated in favour of DatetimeIndex.get_indexer. This method requires a list and therefore is also the go to method if you want to query multiple date times as asked by @fhchl

|

Benjamin Breton · Accepted Answer · 2022-06-06 13:59:28Z

39

DatetimeIndex.get_loc is now deprecated in favour of DatetimeIndex.get_indexer...

ts = pd.to_datetime('2022-05-26 13:19:48.154000')        # example time
iloc_idx = df.index.get_indexer([ts], method='nearest')  # returns absolute index into df e.g. array([5])
loc_idx = df.index[iloc_idx]                             # if you want named index

my_val = df.iloc[iloc_idx]
my_val = df.loc[loc_idx]                                 # as above so below...

edited Jun 6, 2022 at 13:59

Benjamin Breton

1,6071 gold badge18 silver badges46 bronze badges

answered Jun 6, 2022 at 9:20

HarryChil

5795 silver badges10 bronze badges

Comments

Bryan Fok · Accepted Answer · 2017-02-16 06:52:12Z

3

I believe jezrael solution works, but not on my dataframe (which i have no clue why). This is the solution that I came up with.

from bisect import bisect #operate as sorted container
timestamps = np.array(df.index)
upper_index = bisect(timestamps, np_dt64, hi=len(timestamps)-1) #find the upper index of the closest time stamp
df_index = df.index.get_loc(min(timestamps[upper_index], timestamps[upper_index-1],key=lambda x: abs(x - np_dt64))) #find the closest between upper and lower timestamp

answered Feb 16, 2017 at 6:52

Bryan Fok

3,4974 gold badges36 silver badges61 bronze badges

Comments

BugOrFeature · Accepted Answer · 2024-12-01 20:30:43Z

2

in case you are interested in an easy way getting nearest value from a DateTime-indexed DataFrame use Index.asof method:

print(dt)
2016-11-13 22:00:25.450000

dt_indexed = df.index.asof(dt)
print(dt_indexed)
2016-11-13 22:00:28.417561344

edited Dec 1, 2024 at 20:30

BugOrFeature

3291 gold badge6 silver badges14 bronze badges

answered Mar 9, 2024 at 17:44

Anaderi

7915 silver badges8 bronze badges

Comments

Jakob · Accepted Answer · 2020-09-09 09:44:14Z

1

I know it's an old question, but while searching for the same problems as Bryan Fok, I landed here. So for future searchers getting here, I post my solution. My index had 4 non-unique items (possibly due to rounding errors when recording the data). The following worked and showed the correct data:

dt = pd.to_datetime("2016-11-13 22:01:25.450")

s = df.loc[df.index.unique()[df.index.unique().get_loc(dt, method='nearest')]]

However, in case your nearest index occures multiple times, this will return multiple rows. If you want to catch that, you could test for it with:

if len(s) != len(df.columns):
    # do what is appropriate for your case
    # e.g. selecting only the first occurence
    s.iloc[0]

Edit: fixed the catching after some test

edited Sep 9, 2020 at 9:44

answered Sep 8, 2020 at 21:21

Jakob

2462 silver badges7 bronze badges

Collectives™ on Stack Overflow

Pandas DataFrame How to query the closest datetime index?

5 Answers 5

12 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

12 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related