How to filter a list based on elements in another list in python

Question

I have a list A of about 62,000 numbers, and another list B of about 370,000. I would like to filter B so that it contains only elements from A. I tried something like this:

A=[0,3,5,73,88,43,2,1]
B=[0,5,10,42,43,56,83,88,892,1089,3165]
C=[item for item in A if item in set(B)]

Which works, but is obviously very slow for such large lists because (I think?) the search continues through the entire B, even when the element has already been found in B. So the script is going through a list of 370,000 elements 62,000 times.

The elements in A and B are unique (B contains a list of unique values between 0 and 700,000 and A contains a unique subset of those) so once A[i] is found in B, the search can stop. The values are also in ascending order, if that means anything.

Is there some way to do this more quickly?

Patrick Haugh · Accepted Answer · 2019-03-14 20:07:49Z

7

This is creating a new set(B) for every item in A. Instead, use the built-in set.intersection:

C = set(A).intersection(B)

answered Mar 14, 2019 at 20:07

Patrick Haugh

61.4k13 gold badges94 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Chris Over a year ago

Nice! +1. What about set(a) & set(B)? Just seems a bit cleaner.

Patrick Haugh Over a year ago

@ChristianDean That requires a third set to be created, set(B). In the above, only set(A) and C are created as new objects, B is just iterated over.

Kingle Over a year ago

This is extremely fast (and bonus points for simplicity). Exactly what I was looking for. Thank you.

Fukiyel · Accepted Answer · 2019-03-14 20:11:30Z

0

To be really sure what I've done is the fastest possible, I would have done that :

A=[0,3,5,73,88,43,2,1]
B=[0,5,10,42,43,56,83,88,892,1089,3165]

B_filter = B.copy()
C = []
for item in A:
    if filter in B_filter:
        C.append(item)
        B_filter.pop(0) # B_filter is a list, and it's in ascending order so always the first

If you don't care about losing your B list, you can just use B instead of B_filter and not declare B_filter, so you don't have to copy a 370k large list.

answered Mar 14, 2019 at 20:11

Fukiyel

1,1668 silver badges19 bronze badges

Collectives™ on Stack Overflow

How to filter a list based on elements in another list in python

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related