removing duplicate content python

Question

I've got a text file containing multiple jobtitle. I want to remove the title that reoccurs. I created 2 empty array, one for all jobtitle and another which stores non-duplicate values. The code i've used is:

with open('jobtitle.txt') as fp:
jobtitle =[]
jobtitle_original = []
for line in fp:
 jobtitle.append(line)
for i in range(0,len(jobtitle)):
 for j in range(0,len(jobtitle_original)):
  if jobtitle_original[j] == jobtitle[i]:
   continue
  else:
   jobtitle_original.append(jobtitle[i])
print jobtitle_original

But it returns me an empty array. I'm using Python 2.7.

It's not surprising because jobtitle_original is 0 length in the beginning so the inner loop body is never executed. — ElmoVanKielmo
– ElmoVanKielmo, Commented Apr 1, 2014 at 11:27

sshashank124 · Accepted Answer · 2014-04-01 11:26:10Z

1

You can simply use set:

jobs = ['engineer','artist','mechanic','teacher','teacher','engineer','engineer']

print list(set(jobs))
['engineer','artist','mechanic','teacher']

A simpler demonstration:

>>> lst = [1,4,2,4,3,5,3,5,3,5,4,5,4]
>>> print list(set(lst))
[1,4,2,3,5]

set takes a list and creates a set of non-duplicate items. Then, you can simply cast it as a list using list(set(something)).

answered Apr 1, 2014 at 11:26

sshashank124

32.3k10 gold badges72 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ElmoVanKielmo Over a year ago

+1 for you. I've posted my own answer though, just to clarify how to deal with data coming from a file properly.

ElmoVanKielmo · Accepted Answer · 2014-04-01 11:29:17Z

1

Combining your file input and set solution.

with open('jobtitle.txt') as fp:
    result = set(fp.readlines())

answered Apr 1, 2014 at 11:29

ElmoVanKielmo

11.4k2 gold badges35 silver badges51 bronze badges

Collectives™ on Stack Overflow

removing duplicate content python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related