Remove duplicate from the value of particular dictionary key in Python

Something like this:

data = [{'NAME':'John','AGE':23,'NUMBER':345},
        {'NAME':'Michel','AGE':23,'NUMBER':346},
        {'NAME':'RAHUL','AGE':23,'NUMBER':347},
        {'NAME':'Susea','AGE':23,'NUMBER':346},
        {'NAME':'Wincent','AGE':23,'NUMBER':342}]

filtered_data = []
seen = set()
for item in data:
    number = item['NUMBER']
    if number not in seen:
        filtered_data.append(item)
        seen.add(number)
# filtered_data is deduped

Create unique list of Number.

Algo:

1. uni_no is unique number list.
2. data1 final output of filter process.
3. Iterate on data by for loop
4. check if NUMBER is present iterator item.
5. If not then add to data1 and add Number to uni_no

Demo:

>>> data = [{'NAME':'John','AGE':23,'NUMBER':345},
...         {'NAME':'Michel','AGE':23,'NUMBER':346},
...         {'NAME':'RAHUL','AGE':23,'NUMBER':347},
...         {'NAME':'Susea','AGE':23,'NUMBER':346},
...         {'NAME':'Wincent','AGE':23,'NUMBER':342}]
>>> 
>>> data
[{'AGE': 23, 'NAME': 'John', 'NUMBER': 345}, {'AGE': 23, 'NAME': 'Michel', 'NUMBER': 346}, {'AGE': 23, 'NAME': 'RAHUL', 'NUMBER': 347}, {'AGE': 23, 'NAME': 'Susea', 'NUMBER': 346}, {'AGE': 23, 'NAME': 'Wincent', 'NUMBER': 342}]
>>> uni_no = []
>>> data1 = []
>>> for i in data:
...     if i["NUMBER"] not in uni_no:
...         uni_no.append(i["NUMBER"] )
...         data1.append(i)
... 
>>> data1
[{'AGE': 23, 'NAME': 'John', 'NUMBER': 345}, {'AGE': 23, 'NAME': 'Michel', 'NUMBER': 346}, {'AGE': 23, 'NAME': 'RAHUL', 'NUMBER': 347}, {'AGE': 23, 'NAME': 'Wincent', 'NUMBER': 342}]

This works:

key = 'NUMBER'
seen = set()
res = []
for entry in data:
    if not entry[key] in seen:
        res.append(entry)
        seen.add(entry[key])

Result:

>>> res
[{'AGE': 23, 'NAME': 'John', 'NUMBER': 345},
 {'AGE': 23, 'NAME': 'Michel', 'NUMBER': 346},
 {'AGE': 23, 'NAME': 'RAHUL', 'NUMBER': 347},
 {'AGE': 23, 'NAME': 'Wincent', 'NUMBER': 342}]

Use a set to hold the already seen values. For large lists this is much more efficient than using a list such as val in seen_list.

Here are some steps: 1. Create an empty list for your found_numbers. 2. Iterate over all dictionary entries. 3. For each dictionary, look to see if the value for the number key is in your found_numbers list. If the number exists this is a duplicate entry. You can remove it from your list Else the number is not found so this is the first instance of a dictionary with this number. We need to add this number to the found_numbers list.

Note: This assumes a first come first serve attitude towards duplicate entries.