Regex: Finding duplicates in string

Question

I have following regex: \b(\w+) \b(?=.*\b\1)

I need to remove all duplicates in string. So for instance:

Mike Tyson 1. Street 1234 Vietnam ML(12534/97632) Mike Tyson 1234 1. Street Vietnam ML(12534/97632)

should results in:

Mike Tyson 1. Street 1234 Vietnam ML(1234/97632)

I already know why it fails, but I do not know how to fix it. I only look for \w+ and therefore "1." or "ML(156746/615893)" is not beeing found. But when I add these missing characters manually or replace the whole statement by .+ weird stuff is going on.

Can someone help?

anubhava · Accepted Answer · 2020-07-20 13:58:53Z

3

You may use this regex:

(?<!\S)(\S+)\h+(?=(?:\S+\h+)*?\1(?!\S))

Updated RegEx Demo

RegEx Details:

(?<!\S): Lookbehind to assert that we don't have a non-space at previous position
(\S+): Match 1+ non-whitespace character and capture this in group #1
\h+: Match 1+ whitespace
(?=: Start positive lookahead
- (?:\S+\h+)*?: Lazily match 0 more groups where each group consists of 1+ non-whitespace characters followed by 1+ space
- \1: Back reference for group #1
- (?!\S): Must not be followed by a non-whitespace to avoid partial matches
): End positive lookahead

Casimir has made a very good suggestion in comments of using verb (*SKIP) for PCRE flavors as well. This appears to be more efficient as per regex101 website:

~(\S+) \h+ (*SKIP) (?= (?>\S+\h+)*? \1 (?!\S) )~x

edited Jul 20, 2020 at 13:58

answered Jul 20, 2020 at 11:34

anubhava

790k67 gold badges603 silver badges671 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

jor Over a year ago

I have no clue why it works, but I kinda does. Thank you in advance. Could please have look at: regex101.com/r/kBxa7R/6 again? There is just two minor issues left. 1. In the street, the . is replaced even when the number is not the same (represented in line 2 in regex101) 2. For the XX(XXXX/XXXXX) part, it should ony replace if the whole string is the same (represented in line 2 and 3 in regex101)

Vincent Over a year ago

Just for my culture. Can't we just use \b(\S+) instead of (?<!\S)(\S+) ?

Casimir et Hippolyte Over a year ago

Two other versions: regex101.com/r/Npoe1G/1 regex101.com/r/Npoe1G/2

anubhava Over a year ago

Wow, both of them are really good ones @CasimiretHippolyte

anubhava Over a year ago

@Vincent: Please try (?<!\S)(\S+)\h+(?=(?:\S+\h+)*?\1(?!\S)) as per suggestion of Casimir. And yes we can use \b instead of (?<!\S) also but I think \b might be tad slower.

Collectives™ on Stack Overflow

Regex: Finding duplicates in string

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related