Regex to find/remove duplicate line

Question

I have this line, that is sometimes repeated in the html file, and I want to:

1- get a regex to just find files that has that line duplicate

2- get a regex to search and remove the second instance it come on the file, and leave the first. So it keeps only the first, not the second

Given that the lines are not after each other, they are separated with lots of code and text.

The line is:

<script src="/resources/common.js" type="text/javascript"></script>

or it could have words before or after the line that is needed to be removed, like:

<script src="/resources/common.js" type="text/javascript"></script><div id=something"...

I use Notepad++ to search and replace.

Which tool/language are you planning to use for this? Also, is it possible that there may be more than two copies of that line in the file (and if so, do you want to remove all but the first)? — Tim Pietzcker
– Tim Pietzcker, Commented Oct 18, 2012 at 12:36
What tool are you using for regular expressions ? sed ? grep ? java ? an editor ? — Tulains Córdova
– Tulains Córdova, Commented Oct 18, 2012 at 12:41
Notepad++, and yes, it would be great if there are more to remove them all but the first instance in that file.. Thanks — Mike
– Mike, Commented Oct 18, 2012 at 12:41
I was working in a solution using grep and sed, but I see you are using Windows. :( — Tulains Córdova
– Tulains Córdova, Commented Oct 18, 2012 at 12:46

Tim Pietzcker · Accepted Answer · 2012-10-18 17:29:08Z

2

If you were using EditPad Pro (or EditPad Lite, which is free), it would be easy:

Search for

(?s)(?<=<script src="/resources/common\.js" type="text/javascript"></script>.*)<script src="/resources/common\.js" type="text/javascript"></script>

and replace all with nothing.

A screenshot to clarify:

EPP screenshot

With other editors, you will have to apply the following regex repeatedly (once for each duplication):

(?s)(?<=<script src="/resources/common\.js" type="text/javascript"></script>)(.*?)<script src="/resources/common\.js" type="text/javascript"></script>

but this time replace the match with \1.

edited Oct 18, 2012 at 17:29

answered Oct 18, 2012 at 13:04

Tim Pietzcker

337k59 gold badges520 silver badges572 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

dlock Over a year ago

That's a better answer than mine. :)

Mike Over a year ago

Thanks both of you for your time.. I'll try EditPad and see, as I need to do that for thousands of files, and NP++ have an option to search and replace something in a whole directory.. I'll see if EP is fast too..

dlock Over a year ago

@Mike RegexBuddy has built in module for applying these patterns using grep, give it a try too. regexbuddy.com

Tim Pietzcker Over a year ago

@deadlock: Right, but there is no free version of RegexBuddy (but it's worth every penny).

Mike Over a year ago

I tried the regex suggested in the answer using EditPad but it found nothing.. although the file I was testing with had a duplicate.. perhaps because there were some words after the line? I updated the question with another example..

|

dlock · Accepted Answer · 2012-10-18 13:10:42Z

0

You may consider using positive lookbehind which is used for finding and not matching, you can use this to find the first occurrence of your line, then matching the rest of occurrences.

Try This one. It will match all the occurrences of your line except the first one.

(?<=<script src=./resources/common.js..+?</script>.*?)(<script src=./resources/common.js..+?</script>)

note: Positive lookbehinds may or may not work depending on the regex engine you are using, but it should work in most cases.*

More info: Regular Expression Lookaround

edited Oct 18, 2012 at 13:10

answered Oct 18, 2012 at 12:55

dlock

9,5979 gold badges50 silver badges67 bronze badges

4 Comments

Tim Pietzcker Over a year ago

This would only find the first duplicate; also it will only work in .NET languages or EditPadPro, definitely not in Notepad++.

dlock Over a year ago

@TimPietzcker Actually, by trying this pattern on RegexBuddy using JGSoft, it matches all but the first occurrence. And I do not know if this would work on Notepad++. Even if it doesn't, he could use anything else. I think Notepad++ is not the constraint here.

Tim Pietzcker Over a year ago

Yes, but RegexBuddy does use the JGSoft regex engine (like EditPadPro), so you can't generalize the result for other editors. Click on the Use tab and choose Perl to see what I mean.

dlock Over a year ago

@TimPietzcker yes I understand what you mean, that's why I left the note above.

Collectives™ on Stack Overflow

Regex to find/remove duplicate line

2 Answers 2

9 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related