34

I keep bumping into situations where I need to capture a number of tokens from a string and after countless tries I couldn't find a way to simplify the process.

So let's say the text is:

start:test-test-lorem-ipsum-sir-doloret-etc-etc-something:end

This example has 8 items inside, but say it could have between 3 and 10 items.

I'd ideally like something like this:
start:(?:(\w+)-?){3,10}:end nice and clean BUT it only captures the last match. see here

I usually use something like this in simple situations: 

start:(\w+)-(\w+)-(\w+)-?(\w+)?-?(\w+)?-?(\w+)?-?(\w+)?-?(\w+)?-?(\w+)?-?(\w+)?:end

3 groups mandatory and another 7 optional because of the max 10 limit, but this doesn't look 'nice' and it would be a pain to write and track if the max limit was 100 and the matches were more complex. demo

And the best I could do so far:

start:(\w+)-((?1))-((?1))-?((?1))?-?((?1))?-?((?1))?-?((?1))?-?((?1))?:end

shorter especially if the matches are complex but still long. demo

Anyone managed to make it work as a 1 regex-only solution without programming?

I'm mostly interested on how can this be done in PCRE but other flavors would be ok too.

Update:

The purpose is to validate a match and capture individual tokens inside match 0 by RegEx alone, without any OS/Software/Programming-Language limitation

Update 2 (bounty):

With @nhahtdh's help I got to the RegExp below by using \G: 

(?:start:(?=(?:[\w]+(?:-|(?=:end))){3,10}:end)|(?!^)\G-)([\w]+)

demo even shorter, but can be described without repeating code

I'm also interested in the ECMA flavor and as it doesn't support \G wondering if there's another way, especially without using /g modifier.

Improve this question
3
  • Regular expressions are really designed for recognising patterns but you're trying to use this for a changing pattern. You don't say what OS you're on but an Awk (Unix/Linux) or Powershell (Windows) would probably do what you need to do... Commented Mar 7, 2013 at 10:28
  • @RobbieDee: updated post to clarify, looking for a smart way to use RegEx in complex situations without use of any software assistance Commented Mar 8, 2013 at 13:10
  • 1
    @kaᵠ, no, you can't do general things like this in JS with in a single match/step. The only ways to do that are in: .NET (captures repeating group content), or with regex flavors that support \G (or similar API features). Commented Apr 11, 2013 at 20:43

5 Answers 5

Reset to default
37
+50

Read this first!

This post is to show the possibility rather than endorsing the "everything regex" approach to problem. The author has written 3-4 variations, each has subtle bug that are tricky to detect, before reaching the current solution.

For your specific example, there are other better solution that is more maintainable, such as matching and splitting the match along the delimiters.

This post deals with your specific example. I really doubt a full generalization is possible, but the idea behind is reusable for similar cases.

Summary

  • .NET supports capturing repeating pattern with CaptureCollection class.
  • For languages that supports \G and look-behind, we may be able to construct a regex that works with global matching function. It is not easy to write it completely correct and easy to write a subtly buggy regex.
  • For languages without \G and look-behind support: it is possible to emulate \G with ^, by chomping the input string after a single match. (Not covered in this answer).

Solution

This solution assumes the regex engine supports \G match boundary, look-ahead (?=pattern), and look-behind (?<=pattern). Java, Perl, PCRE, .NET, Ruby regex flavors support all those advanced features above.

However, you can go with your regex in .NET. Since .NET supports capturing all instances of that is matched by a capturing group that is repeated via CaptureCollection class.

For your case, it can be done in one regex, with the use of \G match boundary, and look-ahead to constrain the number of repetitions:

(?:start:(?=\w+(?:-\w+){2,9}:end)|(?<=-)\G)(\w+)(?:-|:end)

DEMO. The construction is \w+- repeated, then \w+:end.

(?:start:(?=\w+(?:-\w+){2,9}:end)|(?!^)\G-)(\w+)

DEMO. The construction is \w+ for the first item, then -\w+ repeated. (Thanks to ka ᵠ for the suggestion). This construction is simpler to reason about its correctness, since there are less alternations.

\G match boundary is especially useful when you need to do tokenization, where you need to make sure the engine not skipping ahead and matching stuffs that should have been invalid.

Explanation

Let us break down the regex:

(?:
  start:(?=\w+(?:-\w+){2,9}:end)
    |
  (?<=-)\G
)
(\w+)
(?:-|:end)

The easiest part to recognize is (\w+) in the line before last, which is the word that you want to capture.

The last line is also quite easy to recognize: the word to be matched may be followed by - or :end.

I allow the regex to freely start matching anywhere in the string. In other words, start:...:end can appear anywhere in the string, and any number of times; the regex will simply match all the words. You only need to process the array returned to separate where the matched tokens actually come from.

As for the explanation, the beginning of the regex checks for the presence of the string start:, and the following look-ahead checks that the number of words is within specified limit and it ends with :end. Either that, or we check that the character before the previous match is a -, and continue from previous match.

For the other construction:

(?:
  start:(?=\w+(?:-\w+){2,9}:end)
    |
  (?!^)\G-
)
(\w+)

Everything is almost the same, except that we match start:\w+ first before matching the repetition of the form -\w+. In contrast to the first construction, where we match start:\w+- first, and the repeated instances of \w+- (or \w+:end for the last repetition).

It is quite tricky to make this regex works for matching in middle of the string:

  • We need to check the number of words between start: and :end (as part of the requirement of the original regex).

  • \G matches the beginning of the string also! (?!^) is needed to prevent this behavior. Without taking care of this, the regex may produce a match when there isn't any start:.

    For the first construction, the look-behind (?<=-) already prevent this case ((?!^) is implied by (?<=-)).

  • For the first construction (?:start:(?=\w+(?:-\w+){2,9}:end)|(?<=-)\G)(\w+)(?:-|:end), we need to make sure that we don't match anything funny after :end. The look-behind is for that purpose: it prevents any garbage after :end from matching.

    The second construction doesn't run into this problem, since we will get stuck at : (of :end) after we have matched all the tokens in between.

Validation Version

If you want to do validation that the input string follows the format (no extra stuff in front and behind), and extract the data, you can add anchors as such:

(?:^start:(?=\w+(?:-\w+){2,9}:end$)|(?!^)\G-)(\w+)
(?:^start:(?=\w+(?:-\w+){2,9}:end$)|(?!^)\G)(\w+)(?:-|:end)

(Look-behind is also not needed, but we still need (?!^) to prevent \G from matching the start of the string).

Construction

For all the problems where you want to capture all instances of a repetition, I don't think there exists a general way to modify the regex. One example of a "hard" (or impossible?) case to convert is when a repetition has to backtrack one or more loop to fulfill certain condition to match.

When the original regex describes the whole input string (validation type), it is usually easier to convert compared to a regex that tries to match from the middle of the string (matching type). However, you can always do a match with the original regex, and we convert matching type problem back to validation type problem.

We build such regex by going through these steps:

  • Write a regex that covers the part before the repetition (e.g. start:). Let us call this prefix regex.
  • Match and capture the first instance. (e.g. (\w+))
    (At this point, the first instance and delimiter should have been matched)
  • Add the \G as an alternation. Usually also need to prevent it from matching the start of the string.
  • Add the delimiter (if any). (e.g. -)
    (After this step, the rest of the tokens should have also been matched, except the last maybe)
  • Add the part that covers the part after the repetition (if necessary) (e.g. :end). Let us call the part after the repetition suffix regex (whether we add it to the construction doesn't matter).
  • Now the hard part. You need to check that:
    • There is no other way to start a match, apart from the prefix regex. Take note of the \G branch.
    • There is no way to start any match after the suffix regex has been matched. Take note of how \G branch starts a match.
    • For the first construction, if you mix the suffix regex (e.g. :end) with delimiter (e.g. -) in an alternation, make sure you don't end up allowing the suffix regex as delimiter.
Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

@kaᵠ: Your way is one way, when you have an reasonable upper bound on the number of repetition (such as in your case). Otherwise, you need some extra processing to pick out the correct item. I'd understand if it is not possible to do manipulation on the result, but when it is possible, you should do so.
@kaᵠ: Another point is that, you can always match the whole start:...:end first, then split each of the match along the delimiters. In the case of your example, the delimiter is clear enough that you don't even need my ugly regex solution.
actually after a bit of thought I found that your way is really a viable solution! maybe even better in some cases than capturing all the groups at once! btw: repaired your regex to allow minimum 3 groups check here. please update your answer so I can accept it
@kaᵠ: Thank for the part about 3-10 (that was very careless). But yours also have problem regex101.com/r/zE5hU5. I need probably 10-15 mins to edit.
@Enissay: Please scratch my previous comments. The look-ahead is usually necessary after all. Otherwise, without checking that the whole thing is properly repeated and has a suffix, the regex may match the content even if a suffix is missing.
|
6

Although it might theoretically be possible to write a single expression, it's a lot more practical to match the outer boundaries first and then perform a split on the inner part.

In ECMAScript I would write it like this:

'start:test-test-lorem-ipsum-sir-doloret-etc-etc-something:end'
    .match(/^start:([\w-]+):end$/)[1] // match the inner part
    .split('-') // split inner part (this could be a split regex as well)

In PHP:

$txt = 'start:test-test-lorem-ipsum-sir-doloret-etc-etc-something:end';
if (preg_match('/^start:([\w-]+):end$/', $txt, $matches)) {
    print_r(explode('-', $matches[1]));
}
Improve this answer

1 Comment

+1 Two-step solution is usually much more readable and maintainable.
1

Of course you can use the regex in this quoted string.

"(?<a>\\w+)-(?<b>\\w+)-(?:(?<c>\\w+)" \
"(?:-(?<d>\\w+)(?:-(?<e>\\w+)(?:-(?<f>\\w+)" \
"(?:-(?<g>\\w+)(?:-(?<h>\\w+)(?:-(?<i>\\w+)" \
"(?:-(?<j>\\w+))?" \
")?)?)?" \
")?)?)?" \
")"

Is it a good idea? No, I don't think so.

Improve this answer

Comments

0

Not sure you can do it in that way, but you can use the global flag to find all of the words between the colons, see:

http://regex101.com/r/gK0lX1

You'd have to validate the number of groups yourself though. Without the global flag you're only getting a single match, not all matches - change {3,10} to {1,5} and you get the result 'sir' instead.

import re

s = "start:test-test-lorem-ipsum-sir-doloret-etc-etc-something:end"
print re.findall(r"(\b\w+?\b)(?:-|:end)", s)

produces

['test', 'test', 'lorem', 'ipsum', 'sir', 'doloret', 'etc', 'etc', 'something']

Improve this answer

3 Comments

unfortunately that would find matches in this invalid string: invalid:test-lorem-ipsum-sir-doloret:end besides validating similar strings with less than 3 items and with over 10 items and will also need programming for the matching process
You're trying to do too many things in one go here I think then, it's outside the scope of regular expressions.
RegExes are very powerful! In specific cases you can achieve great results if constructed correctly. See accepted answer^
0

When you combine:

  1. Your observation: any kind of repitition of a single capture group will result in an overwrite of the last capture, thus returning only the last capture of the capture group.
  2. The knowledge: Any kind of capturing based on the parts, instead of the whole, makes it impossible to set a limit on the amount of times the regex engine will repeat. The limit would have to be metadata (not regex).
  3. With a requirement that the answer cannot involve programming (looping), nor an answer that involves simply copy-pasting capturegroups as you've done in your question.

It can be deduced that it cannot be done.

Update: There are some regex engines for which p. 1 is not necessarily true. In that case the regex you have indicated start:(?:(\w+)-?){3,10}:end will do the job (source).

Improve this answer

5 Comments

Actually no it can't. \G requires looping or repetition (p. 3) and will not allow you to set the limit of repetitions (3 to 10) on the regex itself (p. 2).
You can do it with preg_match_all, which makes it quite simple and regex only solution. The capture limit may be possible with look-ahead (in the solution with \G). (I don't claim it works for every cases there are, but there are a class of cases that this works).
Well, it is possible, but not easy to write a correct one. See my answer.
@mrhobo: according to your source it looks like it might be possible to use a simple regex like that in .NET although I am unsure how, tested with regexplanet/dotnet's tool and got the same result (only the last one)
It's easy to do in .NET with named capturing groups. For any particular named group, you can loop over all the matched instances of it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.