Quick way to remove duplicate objects in a List in Python

Question

I have a list of MyClass objects which is made like so:

# The class is MyClass(string_a: str = None, string_b: str = None) 
test_list: List[MyClass] = []
test_clist.append(MyClass("hello", "world"))
test_clist.append(MyClass("hello", ""))
test_clist.append(MyClass("hello", "world"))
test_clist.append(MyClass(None, "world")

I want the end result to only have the 3rd append removed:

# Remove test_clist.append(MyClass("hello", "world"))

This is just a sample and the list of objects can have nothing in the list or n. Is there a way to remove them quickly or a better way like how to quickly tell if it already exists before appending?

try list(set(test_clist)) set cannot have duplicates and are often used to remove them — Nullman
– Nullman, Commented Nov 7, 2021 at 2:49
set() in your example will compare the objects references not values — Ahmed Hany
– Ahmed Hany, Commented Nov 7, 2021 at 3:07
Your type must be hashable, that means you need to implement __hash__(self) and __eq__(self, other) methods — JL0PD
– JL0PD, Commented Nov 7, 2021 at 3:08

Ahmed Hany · Accepted Answer · 2021-11-07 04:18:52Z

2

If your objects are of primitive types, you can use set

list(set(test_clist))

and if not, like your case then you have 2 solutions

1- Implement __hash__() & __eq__()

You have to implement __hash__() & __eq__ in your class in order to use set() to remove the duplicates

see below example

class MyClass(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __repr__(self):
        return f"MyClass({self.x} - {self.y})"

    def __hash__(self):
        return hash((self.x, self.y))

    def __eq__(self, other):
        if self.__class__ != other.__class__:
            return NotImplemented

        return (
            self.x == other.x and
            self.y == other.y
        )

l = []

l.append(MyClass('hello', 'world'))
l.append(MyClass('hello', 'world'))
l.append(MyClass('', 'world'))
l.append(MyClass(None, 'world'))

print(list(set(l)))

Since you have more than one key that you want to use in comparing, __hash__() uses a key tuple.

__repr__() just for representing the class's object as a string.

2- Use 3rd Party Package

check out a package called toolz

then use unique() method to remove the duplicates by passing a key

toolz.unique(test_list, key=lambda x: x.your_attribute)

In your case, you have more than one attribute, so you can combine them into one, for example by creating a concatenated property for them then pass it to toolz.unique() as your key, or just concatenate them on the fly like below

toolz.unique(test_list, key=lambda x: x.first_attribute + x.second_attribute)

edited Nov 7, 2021 at 4:18

answered Nov 7, 2021 at 3:03

Ahmed Hany

1,0707 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2357112 Over a year ago

__eq__ returning False for unrecognized types is usually wrong behavior. Standard practice if an operator overload doesn't recognize the other operand is to return NotImplemented, to give the other operand a chance to handle the operation. (If both objects return NotImplemented from __eq__, == returns False.)

Collectives™ on Stack Overflow

Quick way to remove duplicate objects in a List in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related