Remove duplicate list items accross multiple nested lists

Question

I have the following list:

values = [
    ['registrationController', 'regBean', 'firstName'],
    ['registrationController', 'regBean', 'surname'],
    ['registrationController', 'regBean', 'userName'],
    ['registrationController', 'regBean', 'password'],
    ['registrationController', 'regBean', 'confirmPassword'],
    ['registrationController', 'regBean', 'emailAddress'],
    ['registrationController', 'regBean', 'confirmEmail'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerOne', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerOne', 'answer'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerTwo', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerTwo', 'answer'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerThree', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerThree', 'answer'],
    ['registrationController', 'regBean', 'tAndCAccepted']
]

I am trying to figure out how I can remove values that are preset in all off this lists and keep the values after a certain point when they become uniqe to get something like this:

unique_values = [
    ['firstName'],
    ['surname'],
    ['userName'],
    ['password'],
    ['confirmPassword'],
    ['emailAddress'],
    ['confirmEmail'],
    ['securityQuestionAndAnswerOne', 'question'],
    ['securityQuestionAndAnswerOne', 'answer'],
    ['securityQuestionAndAnswerTwo', 'question'],
    ['securityQuestionAndAnswerTwo', 'answer'],
    ['securityQuestionAndAnswerThree', 'question'],
    ['securityQuestionAndAnswerThree', 'answer'],
    ['tAndCAccepted']
]

Any ideas how I could achive this? I had tried various ways but can't really get close to a working solution.

Yes, ideally as I need to map the values back to the original list they came from. — rtn
– rtn, Commented Oct 31, 2018 at 14:36

user2390182 · Accepted Answer · 2018-10-31 14:37:48Z

4

Use set intersection to get all the common elements and a nested list comprehension to build the cleaned list:

common = set(values[0])
for lst in values[1:]:
    common = common.intersection(lst)

unique_values = [[v for v in lst if v not in common] for lst in values]

answered Oct 31, 2018 at 14:37

user2390182

73.7k6 gold badges71 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Silver · Accepted Answer · 2018-10-31 14:49:08Z

2

Here's an easy to understand and readable version. It will preserve order as well.

common = set.intersection(*values)
reduced_values = [[value for value in l if value not in common] for l in values]

answered Oct 31, 2018 at 14:49

Silver

1,4991 gold badge13 silver badges27 bronze badges

1 Comment

Silver Over a year ago

I see @schobaselgg has already answered with pretty much the same code down to the variable names. But in my defence, I hadn't seen his answer when I posted mine.

Antwane · Accepted Answer · 2018-10-31 14:46:00Z

1

With a list comprehension:

>>> [
    [elt for elt in line if not all(elt in subline for subline in values)]
    for line in values
]

[['firstName'], ['surname'], ['userName'], ['password'], ['confirmPassword'], ['emailAddress'], ['confirmEmail'], ['securityQuestionAndAnswerOne', 'question'], ['securityQuestionAndAnswerOne', 'answer'], ['securityQuestionAndAnswerTwo', 'question'], ['securityQuestionAndAnswerTwo', 'answer'], ['securityQuestionAndAnswerThree', 'question'], ['securityQuestionAndAnswerThree', 'answer'], ['tAndCAccepted']]

edited Oct 31, 2018 at 14:46

answered Oct 31, 2018 at 14:38

Antwane

23.2k8 gold badges56 silver badges91 bronze badges

Comments

Dani Mesejo · Accepted Answer · 2018-10-31 14:47:44Z

-1

Assuming each value is unique within a list you could do something like this:

from collections import Counter
from itertools import chain

values = [
    ['registrationController', 'regBean', 'firstName'],
    ['registrationController', 'regBean', 'surname'],
    ['registrationController', 'regBean', 'userName'],
    ['registrationController', 'regBean', 'password'],
    ['registrationController', 'regBean', 'confirmPassword'],
    ['registrationController', 'regBean', 'emailAddress'],
    ['registrationController', 'regBean', 'confirmEmail'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerOne', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerOne', 'answer'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerTwo', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerTwo', 'answer'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerThree', 'question'],
    ['registrationController', 'regBean', 'securityQuestionAndAnswerThree', 'answer'],
    ['registrationController', 'regBean', 'tAndCAccepted']
]

counts = Counter(chain.from_iterable(values))
result = [[e for e in value if counts[e] != len(values)] for value in values]

print(result)

Output

[['firstName'], ['surname'], ['userName'], ['password'], ['confirmPassword'], ['emailAddress'], ['confirmEmail'], ['securityQuestionAndAnswerOne', 'question'], ['securityQuestionAndAnswerOne', 'answer'], ['securityQuestionAndAnswerTwo', 'question'], ['securityQuestionAndAnswerTwo', 'answer'], ['securityQuestionAndAnswerThree', 'question'], ['securityQuestionAndAnswerThree', 'answer'], ['tAndCAccepted']]

The idea is to count each element across values and get those that do not appear in all the elements of values:

edited Oct 31, 2018 at 14:47

answered Oct 31, 2018 at 14:34

Dani Mesejo

62.2k6 gold badges56 silver badges86 bronze badges

3 Comments

Silver Over a year ago

The question states removing values common to all, not the first two. Maybe that's what he really wants, though.

rtn Over a year ago

Forgot to mention the nested list can be bigger and will not always have a fixed length like the example. picking them out using a slice would work this time but the position where the slice occurs in not guaranteed.

Dani Mesejo Over a year ago

@SilverSlash Updated the answer, now it removes the elements common to all list.

yalex314 · Accepted Answer · 2018-10-31 17:27:48Z

-3

trunc_lists = [tuple(sublist[2:]) for sublist in values]
ans = [list(i) for i in set(trunc_lists)]

Note that the output is un-ordered

edited Oct 31, 2018 at 17:27

answered Oct 31, 2018 at 14:41

yalex314

12 bronze badges

3 Comments

JoeG Over a year ago

What is the "Note"?

sepehr Over a year ago

Welcome to Stack Overflow! Thank you for the code snippet, which might provide some limited, immediate help. A proper explanation would greatly improve its long-term value by describing why this is a good solution to the problem, and would make it more useful to future readers with other similar questions. Please edit your answer to add some explanation, including the assumptions you've made.

yalex314 Over a year ago

It is a typo. I intended to note that the solution would output an un-ordered result

Collectives™ on Stack Overflow

Remove duplicate list items accross multiple nested lists

5 Answers 5

Comments

1 Comment

Comments

3 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

1 Comment

Comments

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related