Remove duplicates using linq

Question

I know this as asked many times but cannot see something that works. I am reading a csv file and then I have to remove duplicate lines based on one of the columns "CustomerID". Basically the CSV file can have multiple lines with the same customerID.

I need to remove the duplicates.

    //DOES NOT WORK
     var finalCustomerList = csvCustomerList.Distinct().ToList();  

     I have also tried this extension method //DOES NOT WORK
     public static IEnumerable<t> RemoveDuplicates<t>(this IEnumerable<t> items)
        {
        return new HashSet<t>(items);
        }

What works for me is

I Read the CSV file into a csvCustomerList

Loop through csvCustomerList and check if a customerExists If it doesnt I add it.

 foreach (var csvCustomer in csvCustomerList)
 {
    var Customer = new customer();
    customer.CustomerID = csvCustomer.CustomerID;
    customer.Name = csvCustomer.Name; 
    //etc.....

    var exists = finalCustomerList.Exists(x => x.CustomerID == csvCustomer.CustomerID);
    if (!exists)
    {
       finalCustomerList.Add(customer);
    }
 }

Is there a better way of doing this?

Daniel Hilgarth · Accepted Answer · 2011-04-05 08:41:05Z

4

For Distinct to work with non standard equality checks, you need to make your class customer implement IEquatable<T>. In the Equals method, simply compare the customer ids and nothing else.
As an alternative, you can use the overload of Distinct that requires an IEqualityComparer<T> and create a class that implements that interface for customer. Like that, you don't need to change the customer class.
Or you can use Morelinq as suggested by another answer.

edited Apr 5, 2011 at 8:41

answered Apr 5, 2011 at 7:15

Daniel Hilgarth

175k41 gold badges346 silver badges450 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user9969 Over a year ago

Thanks that makes sense.However Cannot modify the class

Geoff Appleford · Accepted Answer · 2011-04-05 07:23:08Z

3

For a simple solution, check out Morelinq by Jon Skeet and others.

It has a DistinctBy operator where you can perform a distinct operation by any field. So you could do something like:

var finalCustomerList = csvCustomerList.DistinctBy(c => c.customerID).ToList();

answered Apr 5, 2011 at 7:23

Geoff Appleford

18.9k4 gold badges66 silver badges88 bronze badges

2 Comments

user9969 Over a year ago

Thanks that would work just fine.Not sure I am allowed to ref another third party library

Geoff Appleford Over a year ago

@user231465 - then look at the source and pretend you wrote it yourself. code.google.com/p/morelinq/source/browse/trunk/MoreLinq/… ;-) (only joking of course...)

Collectives™ on Stack Overflow

Remove duplicates using linq

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related