How to find duplicates inside a string?

Question

I want to find out if a string that is comma separated contains only the same values:

test,asd,123,test
test,test,test

Here the 2nd string contains only the word "test". I'd like to identify these strings.

As I want to iterate over 100GB, performance matters a lot.

Which might be the fastest way of determining a boolean result if the string contains only one value repeatedly?

public static boolean stringHasOneValue(String string) {
   String value = null;
   for (split : string.split(",")) {
      if (value == null) {
         value = split;
      } else {
         if (!value.equals(split)) return false;
      }
   }
   return true;
}

The split will end up being a significant bottleneck due to memory allocations if your input is 100GB (especially in JRE7 onwards). Better stick with indexOf. You may not even want to using Strings but instead use input stream or mapped memory through NIO. — billc.cn
– billc.cn, Commented Nov 11, 2015 at 16:11
Is it possible that these entries don't fit into memory? For example, could there be two values, 50 gigs each? — Isaiah van der Elst
– Isaiah van der Elst, Commented Nov 11, 2015 at 16:33

Community · Accepted Answer · 2017-05-23 11:51:56Z

12

No need to split the string at all, in fact no need for any string manipulation.

Find the first word (indexOf comma).
Check the remaining string length is an exact multiple of that word+the separating comma. (i.e. length-1 % (foundLength+1)==0)
Loop through the remainder of the string checking the found word against each portion of the string. Just keep two indexes into the same string and move them both through it. Make sure you check the commas too (i.e. bob,bob,bob matches bob,bobabob does not).
As assylias pointed out there is no need to reset the pointers, just let them run through the String and compare the 1st with 2nd, 2nd with 3rd, etc.

Example loop, you will need to tweak the exact position of startPos to point to the first character after the first comma:

for (int i=startPos;i<str.length();i++) {
   if (str.charAt(i) != str.charAt(i-startPos)) {
      return false;
   }
}
return true;

You won't be able to do it much faster than this given the format the incoming data is arriving in but you can do it with a single linear scan. The length check will eliminate a lot of mismatched cases immediately so is a simple optimization.

edited May 23, 2017 at 11:51

CommunityBot

11 silver badge

answered Nov 11, 2015 at 16:04

Tim B

41.3k16 gold badges88 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Rafael Saraiva Over a year ago

In the third step, you mean to read using indexes right? Since you now know the size of the expected word. As @bill.cn said using the split method is overkill.

Tim B Over a year ago

@RafaelSaraiva Yep, I've just finished editing my answer to clarify that :)

assylias Over a year ago

No need for resetting in step 3 - you can simply compare 2nd occurrence with 3rd occurrence etc.

Jaroslaw Pawlak Over a year ago

Actually, if we are scanning a text file, to know a length we would have to read it whole. I am afraid that this optimisation for checking length, although clever, will not work in this case.

Tim B Over a year ago

@JaroslawPawlak That depends how the incoming data is arriving. If it's a continuous stream then you could well be right. If it's already coming in as Strings in separate lines though (which the question implies it is) then we have the length available.

TR1 · Accepted Answer · 2015-11-11 16:20:49Z

Calling split might be expensive - especially if it is 200 GB data.

Consider something like below (NOT tested and might require a bit of tweaking the index values, but I think you will get the idea) -

public static boolean stringHasOneValue(String string) {

        String seperator = ",";
        int firstSeparator = string.indexOf(seperator); //index of the first separator i.e. the comma
        String firstValue = string.substring(0, firstSeparator); // first value of the comma separated string
        int lengthOfIncrement = firstValue.length() + 1; // the string plus one to accommodate for the comma

        for (int i = 0 ; i < string.length(); i += lengthOfIncrement) {
            String currentValue = string.substring(i, firstValue.length());
            if (!firstValue.equals(currentValue)) {
                return false;
            }
        }

        return true;
    }

Complexity O(n) - assuming Java implementations of substring is efficient. If not - you can write your own substring method that takes the required no of characters from the String.

nafas · Accepted Answer · 2015-11-11 16:26:24Z

0

for a crack just a line code:

(@Tim answer is more efficient)

System.out.println((new HashSet<String>(Arrays.asList("test,test,test".split(","))).size()==1));

answered Nov 11, 2015 at 16:26

nafas

5,4533 gold badges34 silver badges59 bronze badges

Collectives™ on Stack Overflow

How to find duplicates inside a string?

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related