Python, removing duplicates from a list

Question

I suppose the Pythonic way is to use set, which I just learned about. Before I learned about set, I tried the following method which didn't work. Can someone explain what is wrong with this? I got an index out of range error, but I thought the code would never run past the index range for the list as it is written. Please explain what happened.

#!/usr/bin/python
# Remove dupes from a list
def Uniq_list(x):
  for elem in range(len(x)):
    if ( elem == 0 ):
      next
    else:
      if (x[elem] == x[(elem - 1)]):
        x.pop(index(x[elem])
  return x

mylist = ['1', '1', '2', '1', '3', '2']

print Uniq_list(mylist)

since you are popping elemtens from top of the list it gets shorter while you are iterating over it, hence the out of range error — wastl
– wastl, Commented May 24, 2014 at 14:31
There are so many problems in your code. Can you please fix them? — thefourtheye
– thefourtheye, Commented May 24, 2014 at 14:31
next doesn't do anything; it is a no-op as it just references the built-in function. — Martijn Pieters
– Martijn Pieters, Commented May 24, 2014 at 14:31
@wastl That makes sense, thanks a lot! As for the errors in the code, I was trying to go from memory (I have since scrapped this entire function and used set()), so I apologize for the errors, I am a beginner. I was just trying to get my point across as to the index range issue. — GL2014
– GL2014, Commented May 24, 2014 at 14:33

Martijn Pieters · Accepted Answer · 2014-05-24 14:42:52Z

3

The for loop produces the range() object just once, but in the loop you are removing elements from the list, making it shorter.

This leads to another problem: you'll be skipping elements. If you remove element 1, all elements after it shift up one position; element 2 is now element 1, etc. But your loop never takes this into account, so it'll skip the element now moved from position 2 to 1.

This is compounded by the fact you only use the preceding element to test set membership against; the skipped element is suddenly the value against which another value is being tested. They could well be different but that doesn't mean earlier elements are not going to be equal.

Further issues: You used an unnamed index() function in your code which means we cannot verify what it does. If it works like the list.index() method you'd be removing the first occurrence of the value from the list object. This could work, but you already had elem as an index into the list, why search for it again?

The next entry on it's own line is just a reference to the next() function, without ever calling it. As such, it is a no-op. You probably meant to use the continue statement here instead.

A simpler version would have been:

for i in range(len(x), -1, -1):
    if x[i] in x[:i]:
        del x

e.g. starting from the end of the list, if the current element is present in the list before this position, remove it. However, using a set would be far more efficient still.

edited May 24, 2014 at 14:42

answered May 24, 2014 at 14:33

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

GL2014 Over a year ago

Thanks very much, I will accept when time limit allows me to do so.

GL2014 Over a year ago

My usage of next was intended as a method to skip testing x[0] against x[0 - 1]. I wanted a noop in the sense that I wanted to jump to the next iteration in the loop if elem evaluates to 0. Is that not what it would be doing?

Martijn Pieters Over a year ago

@GreggLeventhal: No, that's what the continue statement does, though.

Martijn Pieters Over a year ago

@GreggLeventhal: it is not needed here, really; the else branch is never entered if the if statement was picked. You could just have done if elem != 0: ... instead.

GL2014 Over a year ago

Darn, I wasn't sure if it was next or continue. Thanks again!

Collectives™ on Stack Overflow

Python, removing duplicates from a list

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related