How can I speed up a loop that queries a kd-tree?

Question

The following section of my code is taking ages to run (it's the only loop in the function, so it's the most likely culprit):

tree = KDTree(x_rest)
for i in range(len(x_lost)):
    _, idx = tree.query([x_lost[i]], k=int(np.sqrt(len(x_rest))), p=1)
    y_lost[i] = mode(y_rest[idx][0])[0][0]

Is there a way to speed this up? I have a few suggestions from Stack Overflow:

This answer suggests using Cython. I'm not particularly familiar with it, but I'm not very against it either.
This answer uses a multiprocessing Pool. I'm not sure how useful this will be: my current execution takes over 12h to run, and I'm hoping for at least a 5-10x speedup (though that may not be possible).

I can't believe I didn't think of that. So would calling tree.query(x_lost, k=...) give me an array of idx? Perhaps the last line could use a list comprehension then. — Rahul
– Rahul, Commented Feb 5, 2023 at 0:15

Nick ODell · Accepted Answer · 2023-02-05 17:32:35Z

2

Here are a few notes about how you could speed this up:

This code loops over x_rest, and calls tree.query() with one point from x_rest at a time. However, query() supports querying multiple points at once. The loop inside query() is implemented in Cython, so I would expect it to be much faster than a loop written in Python. If you call it like this, it will return an array of matches.
The query() function supports a parameter called workers, which if set to a value larger than one, runs your query in parallel. Since workers is implemented using threads, it will likely be faster than a solution using multiprocessing.Pool, since it avoids pickling. See the documentation.
The code above doesn't define the mode() function, but I'm assuming it's scipy.stats.mode(). If that's the case, rather than calling mode() repeatedly, you can use the axis argument, which would let you take the mode of nearby points for multiple queries at once.

edited Feb 5, 2023 at 17:32

answered Feb 5, 2023 at 17:27

Nick ODell

28.1k7 gold badges52 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rahul Over a year ago

This is exactly what I did! I rewrote the function in Cython, moved the query to a batch call, and used scipy.stats.mode with axis=1. I couldn't set workers sadly, since the computer cluster I'm using is on an older version of scipy, but doing this got me a 50x speedup, which was enough.

Collectives™ on Stack Overflow

How can I speed up a loop that queries a kd-tree?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related