Regex works on regex101, but not in powershell... why?

Question

I have this test data:

^Test data



This is all just test data 


testing 123
ABC>space "ABC"

ABC>

And I've setup a regex on regex101.com (^\^|ERROR).*((|\n|\r|\w|\W)?)+(?=ABC>)

The expression is returning just what I want on the site:

I am using this powershell I wrote to get content similar to the content above and looping through files, and looking for matches of the same regex expression.

$files = gci "\\server\path"
$content = @()


ForEach($file in $files){
    # Set script name
    $scriptname = "ABC TEST_081722"

    # Get the name of the task for the logfile's filename. 
    $taskname = "THIS IS A TEST!!!" 

    # Create log file with a datestamp MMDDYY
    $datestamp = (get-date).ToString('MMddyy')
    $logfilepath = "\\server\path\Logs\$($taskname)\$($file.basename).log"
    $log_dir = "\\server\path\Logs\$($taskname)\"

    # Get the content of the log file. We are only interested in getting lines which match a regex for our command line and our output line. 
    $content_raw = get-content $logfilepath -raw

    $content_raw -match "(^\^|ERROR).*((|\n|\r|\w|\W)?)+(?=ABC>)"
    
    Write-host -f yellow $file.fullname
    $matches
    $matches.clear()

                                                                        
    start-sleep -s 2
}

The regex finds a match in two of my three test files, but not the first one which has the exact same string content as my example above. Why does it find a match in the 2nd and 3rd file but not the first?

The content of the 2nd and 3rd file are like so

ABC>W !,MSG

ERROR^BATCH~Batch in use
ABC>space "ABC"

So these two files do not have a line starting with a "^" symbol. It starts with "ERROR" which I accounted for with my OR statement in my regex. I just don't understand how it is able to find the lines which start with "ERROR" find, but not finding the lines from the first file which starts with "^" carat.

Oh I see. The first file would work without the blank line one. It would be nice if that were text instead of a picture. Select-string is nice in powershell 7 since it highlights the match. — js2010
– js2010, Commented Aug 18, 2022 at 16:50

mklement0 · Accepted Answer · 2022-08-18 15:26:30Z

7

Try the following regex instead:

(?sm)(?:^\^|^ERROR).*?(?=\r?\nABC>)

^{Note: This is a streamlined, working version of your regex (without capture groups); a corrected form of your regex would be (?m)(^\^|^ERROR).*((|\n|\r|\w|\W)?)+(?=ABC>), for the reasons explained below.}

Note that PowerShell's operators are case-insensitive by default (as is PowerShell in general). For case-sensitivity, use the c-prefixed operator variants, i.e., -cmatch in this case.

See this regex101.com page, where you can experiment with text from your files interactively.

As for what you tried:

^ only matches at the start of individual lines if the MultiLine regex option is in effect, which you can activate using inline syntax with (?m) - note that, unlike PowerShell, regex101.com has this option turned on by default (see the option letters such as gm to the right of the regex input field), which would explain why you didn't see the problem there.
- Similarly, (?s) activates the SingleLine regex option, which makes . match newline characters (\n) too.
^\^|ERROR applies the start-of-input/line ^ metacharacter only to the (escaped) ^ character, not also to ERROR on the other side of the alternation (|).
- Your test data wasn't at the very start of your input file (as shown in the screenshot), causing the ^ to be ineffective in the absence of (?m).
- Conversely, because substring ERROR was (accidentally) not anchored, it still matched (but would match anywhere on a line).

Note:

As of this writing, regex101.com has no dedicated PowerShell support, and the closest approximation, .NET (C#), has defaults that are at odds with PowerShell's defaults.
For guidance on how to use regex101.com with PowerShell, including a link to a feature request to introduce PowerShell support in the future, see this answer.

edited Aug 18, 2022 at 15:26

answered Aug 18, 2022 at 0:50

mklement0

452k68 gold badges728 silver badges988 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

RavinderSingh13 Over a year ago

This is great and very informative answer, thanks for sharing. Just curious if powershell regex engine is not supported in regex101 site, so do we have any other online site for demo OR its only we need to test it as a .ps1 program only to test it, just wanted to know, Thank you.

mklement0 Over a year ago

Thanks, @RavinderSingh13. PowerShell doesn't have its own regex engine, it uses .NET's, just like C# (or any .NET language). However, (a) PowerShell uses different defaults ((?i), i.e. case-INsensitivity is implied), (b) regex101.com assumes (?m) by default, which isn't even the .NET engine's default (global matching is also assumed, and whether that applies depends on which specific .NET method / PowerShell operator you use), and (c) rege101.com assumes that the regex is specified as a C# verbatim string, whose escaping rules differ from single-quoted PowerShell strings.

RavinderSingh13 Over a year ago

This is very informative @mklement0 Thank you for letting know, cheers.

mklement0 Over a year ago

Glad to hear it, @RavinderSingh13. To summarize: You can already make regex101.com work with PowerShell, but (a) you need to (first select flavor .NET (C#) and) adjust the default matching options and (b) if your '...'-enclosed PowerShell regex contains " or ', you need to modify it to satisfy the syntax of a C# verbatim string. Hopefully, the feature request to introduce specific PowerShell support, if implemented, will make that unnecessary. I suggest up-voting it, if you too would like to see it implemented.

shadow2020 Over a year ago

This does not explain why it finds the regex if the line starts with "ERROR" but does not if the line starts with carot. I fail to understand how case sensitivity would cause it to fail on a carot.

|

Collectives™ on Stack Overflow

Regex works on regex101, but not in powershell... why?

1 Answer 1

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related