Find Duplicate Objects in JSON Array

Question

I am relatively new to c# and need some guidance. I need to search through a large json array to find duplicate objects and list how many occurrences.

Sample Array

{
    "data": 
        [
          {
            "date": "20220101",
            "someID": "ID1",
            "someType1": "SRVC",
            "someType2": "SEND"
          },
          {
            "date": "20220101",
            "someID": "ID1",
            "someType1": "SRVC",
            "someType2": "SEND"
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "RECV"
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "RECV"
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "RECV"
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "SEND"
          },          
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "SEND"
          },          
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "SEND"
          }
          
        ]
}

I have been able to search through and find duplicate elements but not the complete object. The output should be:

{
    "result": 
        [
          {
            "date": "20220101",
            "someID": "ID1",
            "someType1": "SRVC",
            "someType2": "SEND",
            "objCount": 2
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "RECV",
            "objCount": 3
          },
          {
            "date": "20220101",
            "someID": "ID2",
            "someType1": "SRVC",
            "someType2": "SEND",
            "objCount": 3
          }
          
        ]
}

Any help to point me in the right direction would be great. Thanks in advance

You can use a dictionary to add distinct "key" and in the value you can update the occurence — Joe Taras
– Joe Taras, Commented Feb 9, 2022 at 15:02

D-Shih · Accepted Answer · 2022-02-09 15:20:21Z

3

You can try to create models to let your JSON data be objects

public class Datum
{
    public string date { get; set; }
    public string someID { get; set; }
    public string someType1 { get; set; }
    public string someType2 { get; set; }
}

public class Root
{
    public List<Datum> data { get; set; }
}

then you can use JsonConvert.DeserializeObject method to deserialize the JSON value to be objects, then use lambda to get duplicate values from the list.

var root = JsonConvert.DeserializeObject<Root>(data);
var res = JsonConvert.SerializeObject(root.data.GroupBy(x=> new {
    x.someID,
    x.someType1,
    x.someType2,
    x.date
}).Where(x=>x.Count()>1).Select(x=>x.Key));

c# online

answered Feb 9, 2022 at 15:20

D-Shih

46.5k6 gold badges38 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

T.S. · Accepted Answer · 2022-02-09 17:04:49Z

Step 1 would be declaring a model

public class Data
{
    public DateTime date { get; set; }
    public int someID { get; set; }
    public string someType1 { get; set; }
    public string someType2 { get; set; }
}

Step 2 - deserialize JSON, in this case, using Newtonsoft

var dataList = JsonConvert.DeserializeObject<Data>(json);

Step 3 - produce result

var output = dataList.GroupBy(m => (m.date, m.someID, m.someType1, m.someType2))
    .Select(g => new 
        {
            date = g.Key.date,
            someID = g.Key.someID,
            someType1 = g.Key.someType1,
            someType2 = g.Key.someType2,
            objCount = g.Count()
        });

2 things - potentially LINQ syntax is different for different versions of c#. And you might need to declare a special model for the output.

Monsieur Merso · Accepted Answer · 2022-02-09 16:25:24Z

Another approach, with usage of anonymous classes:

using JsonDocument jsonDocument = JsonDocument.Parse(jsonString);

var output = jsonDocument.RootElement
    .GetProperty("data") // get root level "data" property
    .EnumerateArray()
    // create anonymous type on which C# will group list
    // this will work, because C# generates custom field comparing 
    // Equals for anonymous types
    .GroupBy(e => new { 
            date = e.GetProperty("date").GetString(),
            someID = e.GetProperty("someID").GetString(),
            someType1 = e.GetProperty("someType1").GetString(),
            someType2 = e.GetProperty("someType2").GetString(),
    }).Select(g => new {
            date = g.Key.date,
            someID = g.Key.someID,
            someType1 = g.Key.someType1,
            someType2 = g.Key.someType2,
            objCount = g.Count(),
    });

Collectives™ on Stack Overflow

Find Duplicate Objects in JSON Array

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related