C# How to filter a list and remove duplicates?

Question

I have a List of Type X. This contains fields and I need to return only unique records from the list. I need to use one of the field/property (OIndex) that contains a timestamp and filter it using that property. List is like this:

> 2c55-Checked-branchDeb-20160501121315-05
> 2c60-Checked-branchDeb-20160506121315-06
> 2c55-Checked-branchDeb-20160601121315-07
> 2c55-Checked-branchDeb-20160601141315-07
> 2c60-Checked-branchDeb-20160720121315-08

In the example above the last field is the recordId so we have a duplicate record of "07". The timestamp is field four. So I want to get the all the records except that 3rd which is a duplicate. The latest version of record "07" is the fourth line.

I started doing the code but struggling. So far:

List<X> originalRecords = GetSomeMethod(); //this method returns our list above

var duplicateKeys = originalRecords.GroupBy(x => x.Record)  //x.Record is the record as shown above "05", "06" etc
                        .Where(g => g.Count() > 1)
                        .Select(y => y.Key);

What do I do now? Now that I have the duplicate keys. I think I need to go through the OriginalRecords list again and see if it contains the duplicate key. And then use substring on the datetime. Store this somewhere and then remove the record which is not the latest. And save the original records with the filter. Thanks

Can you define the keys on this: 2c55-Checked-branchDeb-20160501121315-05? — Blue
– Blue, Commented Aug 11, 2016 at 17:09

Sergey Kalinichenko · Accepted Answer · 2016-08-11 17:46:28Z

10

You don't need to find duplicate keys explicitly, you could simply select first from each group:

var res == originalRecords
    .GroupBy(x => x.RecordId)
    .Select(g => g.OrderByDescending(x => x.DateTimeField).First());

There is no field for datetimefield as in your code. I simply have a string field which contains the datetime together with other data. The record however has a Record Id field.

You can split your records on a dash, grab the date-time portion, and sort on it. Your date/time is in a format that lets you sort lexicographically, so you can skip parsing the date.

Assuming that there are no dashes, and that all strings are formatted in the same way, x.TextString.Split('-')[3] expression will give you the timestamp portion of your record:

var res == originalRecords
    .GroupBy(x => x.RecordId)
    .Select(g => g.OrderByDescending(x => x.TextString.Split('-')[3]).First());

edited Aug 11, 2016 at 17:46

answered Aug 11, 2016 at 17:09

Sergey Kalinichenko

729k85 gold badges1.2k silver badges1.6k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Blue Over a year ago

He should order by the datetime field first (Decending), to make sure g.First() is pulling the latest field.

Sergey Kalinichenko Over a year ago

@FrankerZ Thank you!

user2906420 Over a year ago

@dasblinkenlight the above code will NOT work as the records are not from a table or there is no field for datetimefield as in your code. I simply have a string field which contains the datetime together with other data. The record however has a Record Id field. hope it makes sense.

Sergey Kalinichenko Over a year ago

@user2906420 Please take a look at the edit. I removed Distinct as irrelevant, because it looks like you need a specific duplicate.

user2906420 Over a year ago

@dasblinkenlight thanks for the edit, I will give it a go and get back

guijob · Accepted Answer · 2016-08-11 17:18:36Z

0

This should solve your problem:

List<X> originalRecords = GetSomeMethod();
Dictionary<int, X> records = new Dictionary<int, X>();

foreach (X record in originalRecords) {

    if(records[record.recordId] != null) {
        if(records[record.recordId].stamp < record.stamp){
            records[record.recordId] = record;
        }
    }
    else {
        records[record.recordId] = record;
    }
}

Your answer are records.Values

Hope it helps

edited Aug 11, 2016 at 17:18

answered Aug 11, 2016 at 17:16

guijob

4,4883 gold badges23 silver badges41 bronze badges

3 Comments

Blue Over a year ago

Where exactly is recordId being set in this line: if(record[recordId] != null) {?

guijob Over a year ago

Is was wrong.. record.recordId is the object's last field.. I forgot to declare a dict either.. I updated my post

user2906420 Over a year ago

@J.Guilherme thanks for your help but the record is a string which has timestamp in it. I really need to use substring and indexOf and extract the timestamp and then compare with each other.

Collectives™ on Stack Overflow

C# How to filter a list and remove duplicates?

2 Answers 2

5 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related