How to find duplicate elements in a Stream in Java

Question

I'm trying to find duplicate entries in map values. But the thing is the list of values have multiple attributes/properties. Basically, if a title shows up more than once in a database, I would mark one entry as unique and mark the rest as duplicates.

Here's my current code:

// I have a Map that looks like...

host1 : id | title | host1 | url | state | duplicate
        id | title | host1 | url | state | duplicate

host2 : id | title | host2 | url | state | duplicate
        id | title | host2 | url | state | duplicate


    for (Map.Entry<String, List<Record>> e : recordsByHost.entrySet()) {
      boolean executed = false;
      for (Record r : e.getValue()) {
        int frequency = Collections.frequency(
          e
            .getValue()
            .stream()
            .map(Record::getTitle)
            .collect(Collectors.toList()),
          r.getTitle()
        );
        if ((frequency > 1) && (!executed)) {
          markDuplicates(r.getId(), r.getTitle());
          executed = true;
        } else {
          executed = false;
        }

The issue is when frequency is more than 2 (three records with the same title), the line evaluates to false and treats the third record / second duplicate as "unique".

I've been trying to rework my logic but I'm afraid I'm stuck. Any help / suggestions to get me unstuck would be greatly appreciated.

Do you absolutely have to use a Stream? I think responding to the boolean value returned from Set.add would be more useful. — VGR
– VGR, Commented Aug 4, 2021 at 18:42
@VGR - Not necessarily. What exactly do you mean by "responding to the boolean value returned from Set.add"? Do you have helpful articles you can share? — jayteezer
– jayteezer, Commented Aug 4, 2021 at 18:54

VGR · Accepted Answer · 2021-08-04 18:58:57Z

4

Set.add (and in fact, Collection.add) returns true if and only if the value was actually added to the Set. Since a Set always enforces uniqueness, you can use this to find duplicates:

void markDuplicates(Iterable<? extends Record> records) {
    Set<String> foundTitles = new HashSet<>();

    for (Record r : records) {
        String title = r.getTitle();
        if (title != null && !foundTitles.add(title)) {
            // title was not added, because it's already been found.
            markAsDuplicate(r);
        }
    }
}

answered Aug 4, 2021 at 18:58

VGR

44.9k4 gold badges52 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jayteezer Over a year ago

Learn something new everyday! Thanks for that! I didn't exactly use all the code you provided but I am using set.add() instead of boolean = executed and all my unit tests work now! :)

Collectives™ on Stack Overflow

How to find duplicate elements in a Stream in Java

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related