Python: remove duplicate items from a list while iterating

Question

I have a list named results, I want to get rid of the duplicate items in the list, the expected result and my results does not match, I checked my code but cannot find what is the problem, what happened? Thanks a lot!

results = [[-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 
    6], [-4, -2, 6], [-4, 0, 4], [-4, 0, 4], [-4, 1, 3], [-4, 1, 3], [-4, 2, 
    2], [-4, 2, 2], [-4, 2, 2], [-2, -2, 4], [-2, -2, 4], [-2, -2, 4], [-2, 
    -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4], [-2, -2, 4], 
    [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2]]

i = 0
while i < len(results):  
    j = i+1
    while j < len(results):
        if(set(results[i]) == set(results[j])):
            results.remove(results[j])
        else:
            j = j+1
    i = i+1
print(results)

OUTPUT:
[[-4,-2,6],[-4,0,4],[-4,1,3],[-4,2,2],[-2,-2,4],[-2,-2,4],[-2,0,2]]

EXPECTED RESULT:
[[-4,-2,6],[-4,0,4],[-4,1,3],[-4,2,2],[-2,-2,4],[-2,0,2]]

UPDATE: I got it. no problem with the logic of this code but I made a simple mistake with one place (sorry...I am a newbie). I should replace method "remove" by method "del", because I want to remove the item with specified index, if use "remove", it always remove the first one shows up in the list of that value. Anyway, Thanks to all!

For example:
myList = ['Apple', 'Banana', 'Carrot','Apple']
myList.remove(myList[3])
print(myList)
expected output:['Apple', 'Banana', 'Carrot']
actual output: ['Banana', 'Carrot', 'Apple']

myList = ['Apple', 'Banana', 'Carrot','Apple']
del (myList[3])
print(myList)
OUTPUT: ['Apple', 'Banana', 'Carrot']

SOLUTION to my question:

### use "del" instead of "remove"
#results.remove(results[j])
del results[j]

Another simple test example similar to my original question:
results = [[-2, -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4]]
i = 0
while i < len(results):  
    j = i+1
    while j < len(results):
        print(results[i],results[j])
            if(set(results[i]) == set(results[j])):
                #would get incorrect answer with "replace"
                results.remove(results[j])
                #try "del" to get the correct answer
                #del (results[j]) 
    else:
        j = j+1
i = i+1
print(results)

You're iterating over an object while mutating it, which can be tricky. Try iterating over it backwards. — Jared Goguen
– Jared Goguen, Commented Apr 20, 2017 at 20:34
Do not remove from existing list, it will be better to create new list and add unique items to it. — Uladzimir Palekh
– Uladzimir Palekh, Commented Apr 20, 2017 at 20:34
If you really want to understand what is wrong with your code, it seems to me that setting j to i+1 is the mistake here. You want j to take all values but i, so j must go from 0 to your length and skip only current i value. Now as the other answers state, you have more pythonic ways to achieve duplicates removal. — Tanguy A.
– Tanguy A., Commented Apr 20, 2017 at 20:44

Community · Accepted Answer · 2017-05-23 12:17:42Z

3

Thanks to Remove duplicate sublists from a list

results = [[-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 
    6], [-4, -2, 6], [-4, 0, 4], [-4, 0, 4], [-4, 1, 3], [-4, 1, 3], [-4, 2, 
    2], [-4, 2, 2], [-4, 2, 2], [-2, -2, 4], [-2, -2, 4], [-2, -2, 4], [-2, 
    -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4], [-2, -2, 4], 
    [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2]]

results = [list(x) for x in set([tuple(x) for x in results])]
print (results)

prints:

[[-4, -2, 6], [-2, 0, 2], [-4, 1, 3], [-2, -2, 4], [-4, 2, 2], [-4, 0, 4]]

edited May 23, 2017 at 12:17

CommunityBot

11 silver badge

answered Apr 20, 2017 at 20:36

ragardner

2,0056 gold badges26 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Xuyong Over a year ago

thank you ,this may be an easier way to do it. BTW, I find the problem with my own code.

Jared Goguen · Accepted Answer · 2017-04-20 20:38:22Z

3

The following loop will iterate backwards over the pairs, allowing you to mutate the list during iteration.

i = len(results) - 1
while i >= 1:  
    j = i - 1
    while j >= 0:
        if(set(results[i]) == set(results[j])):
            results.remove(results[j])
            i -= 1
        j -= 1
    i -= 1

But, yes, using sets would probably be better. A functional version of the above would be:

map(list, set(map(tuple, results)))

answered Apr 20, 2017 at 20:38

Jared Goguen

9,0182 gold badges22 silver badges39 bronze badges

1 Comment

Xuyong Over a year ago

it is interesting to do backwards to avoid the problem, your answer may be most near to what I want. thank you ,BTW, I find the problem with my own code.

Community · Accepted Answer · 2017-05-23 12:34:15Z

2

Set-ify everything.

Convert the inner lists to sets to remove duplicate elements within them. Push all these sets into another set to get rid of duplicate (sub)sets. Finally, convert everything back to a list of lists:

no_dups = [list(s) for s in set(frozenset(l) for l in results))]

I refered to How can I create a Set of Sets in Python? to figure out the bit about frozenset. You can't just convert the sublists to sets since the outer set needs immutable elements to hash (and regular sets are mutable). Of course, you could get around this by using maps/tuples as other answers point out.

edited May 23, 2017 at 12:34

CommunityBot

11 silver badge

answered Apr 20, 2017 at 20:39

sgrg

1,2609 silver badges15 bronze badges

Comments

Moinuddin Quadri · Accepted Answer · 2017-04-20 20:46:53Z

1

You may use set with map as:

my_uniques = set(map(tuple, results))
#                     ^ type-cast inner list to tuples
#                       because lists are not hashable and
#                       hence can't be used with set

where my_uniques will be a set of unique tuple objects:

set([(-2, -2, 4), (-4, 0, 4), (-4, -2, 6), (-4, 2, 2), (-2, 0, 2), (-4, 1, 3)])

If you want to convert it to list of lists (which I don't think is necessary), you have to explicitly do:

my_uniques  = list(map(list, my_uniques))

answered Apr 20, 2017 at 20:46

Moinuddin Quadri

48.4k13 gold badges101 silver badges138 bronze badges

2 Comments

Xuyong Over a year ago

thank you , I tried several test cases, but sometimes it still has duplicate item, I did some change to my original code and it works. anyway, thanks very for your help, your method could be a easier way to achieve my goal.

Moinuddin Quadri Over a year ago

I strongly doubt about the duplicate tuples in the set. But it is worth mentioning that tuple (0,0,1) and (1,0,0) are different i.e. order matters. In case for you this order doesn't matter, you should be using sorted() function on each tuple which will remove the tuples with same number but with different positions from the set object

Collectives™ on Stack Overflow

Python: remove duplicate items from a list while iterating

4 Answers 4

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related