linux sed expression to select text between markers

Question

Here is a challenge for regex gurus. Need a very simple sed expression to select text between markers.

Here is an example text. Please mind it can contain any special chars, TABS and white spaces even though this example doesn't depict all possible combinations.

^[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~

Select text between first matched start of marker M1 to last matched end of marker M3. The text to select from example is

bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

Select text between last matched start of marker M1 to first matched end of marker M3. The text to select from example is

ccc[$cM2ddddM2eeeee

I tried this but it select last start of marker to last end of marker

echo "^[[200~a^[[200~aaa aM1bb bbbM1ccc[\$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~"|sed -E "s|.*M1(.*)M3.*$|\1|g"

ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

How it is possible? single sed regex expression would be the best. What I mean single regex is one for each above two requirements. i.e. two regex Also need the equivalent python re expression.

THe regular expression is probably the same for any regex engine. The key here is the greediness of the partial matches. — LMC
– LMC, Commented Mar 27 at 1:52
Is guaranteed that the wanted appearance of the start marker will appear before the wanted appearance of the end marker? — John Bollinger
– John Bollinger, Commented Mar 27 at 3:15
yes thats right - wanted appearance of the start marker will appear before the wanted appearance of the end marker — Ramanan T
– Ramanan T, Commented Mar 27 at 3:43

markp-fuso · Accepted Answer · 2025-03-27 02:22:12Z

4

If you don't mind a compound sed script ... one approach for non-greedy matches is to change a delimiter to a string that does not occur in the input and then drive the match off of that replacement character.

While OP has stated the input could contain any special characters I'm going to assume the input cannot contain the NUL character (0x00).

Tweaking OP's current sed script ...

For OP's longer match:

$ echo "^[[200~a^[[200~aaa aM1bb bbbM1ccc[\$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~" | sed -E "s|M1|\x00|1; s|.*\x00(.*)M3.*$|\1|g"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

Where:

s|M1|\x00|1 - convert the first M1 to NUL
s|.*\x00(.*)M3.*$|\1|g - search for string between NUL and M3
since we've removed the NUL there's no need to convert a NUL back to M1

For OP's shorter match:

$ echo "^[[200~a^[[200~aaa aM1bb bbbM1ccc[\$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~" | sed -E "s|M3|\x00|1; s|.*M1(.*)\x00.*$|\1|g"
ccc[$cM2ddddM2eeeee

Where:

s|M3|\x00|1 - convert the first M3 to NUL
s|.*M1(.*)\x00.*$|\1|g - search for string between M1 and NUL`
since we've removed the NUL there's no need to convert a NUL back to M3

NOTE: I don't work with python but since these are (relatively) simple sed regexes I'm guessing they should be (relatively) easy to migrate to python ... ???

edited Mar 27 at 2:22

answered Mar 27 at 2:08

markp-fuso

38.6k5 gold badges24 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ramanan T Mar 28 at 0:19

if it has the python solution I would have accepted this. This is the first in this kind for sed

markp-fuso Mar 28 at 0:57

@RamananT are you saying this relatively simple sed script is not (easily) implemented in python?

Ramanan T Mar 28 at 1:06

Not really but if it is going to be the accepted answer then prefer to have that as complete solution. It would help others like me. Normally I always look for the accepted answer in any post.

markp-fuso Mar 28 at 13:53

well, as I stated in the answer, I don't use python so I'm not in a position to add a python solution to my answer; if you have this solution working in python you could add your own answer, or update the question with an addendum stating something like ... took markp-fuso's answer for bash and converted to the following python equivalent

Renaud Pacalet · Accepted Answer · 2025-03-28 06:11:30Z

3

The second case is easy, even with sed:

$ a='^[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~'
$ sed -E 's/.*M1|M3.*//g' <<< "$a"
ccc[$cM2ddddM2eeeee

The first case is more complex because of the greediness of sed regexes. If you can use python or perl, instead of sed, you can harness their non-greedy .*? operator:

$ python -c 'import sys,re; print("\n".join(re.sub(r".*?M1|M3.*?","",l) for l in sys.stdin),end="")' <<< "$a"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh
$ perl -pe 's/.*?M1|M3.*?//g' <<< "$a"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

A bit shorter with python if you have only one line of text to process and if we pass it as an argument:

$ python -c 'import sys,re; print(re.sub(r".*?M1|M3.*?","",sys.argv[1]))' "$a"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

With sed, one possibility consists in first inserting separator characters that do not appear in the input string, for instance newlines, and then keeping only what appears between them. If your sed supports \n for newline in the replacement string of the substitute command:

$ sed -E 's/M1(.*)M3/\n\1\n/;s/.*\n(.*)\n.*/\1/' <<< "$a"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

Else, with any sed:

$ sed -E 's/M1(.*)M3/\
\1\
/;s/.*\n(.*)\n.*/\1/' <<< "$a"
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

Note: as your shell is bash, if you absolutely want a one-liner you can use a $'...' character sequence:

$ sed -E $'s/M1(.*)M3/\\\n\\1\\\n/;s/.*\\n(.*)\\n.*/\\1/' <<< "$a"

edited Mar 28 at 6:11

answered Mar 27 at 5:34

Renaud Pacalet

30.7k3 gold badges42 silver badges60 bronze badges

2 Comments

Ramanan T Mar 28 at 0:23

Do you think you can clarify or make the python solution clear or simpler. sed solution is pretty good and don't require any further improvement.

Renaud Pacalet Mar 28 at 6:10

Unfortunately python is not the best tool for one-liners. You must import modules, explicitly read the standard input, print... perl has the nice -p option that makes all this implicit but up to my knowledge there is no python equivalent. If you have only one line of text to process we can maybe pass it as a parameter and save a few characters (see my updated answer). I am not a python expert, so there may be a shorter solution.

John Bollinger · Accepted Answer · 2025-03-27 05:22:18Z

How it is possible? single sed regex expression would be the best.

Presumably you mean you would prefer a single s command, as that's what you present in your example. A regex is one part of an s command, but the whole command is more than just a regex.

Unlike some other regex dialects, POSIX regular expressions (the kind used by sed) have no non-greedy, arbitrary-count quantifiers. That makes it a bit tricky to satisfy a requirement to match up to the first appearance of a multi-character substring when there may be more than one appearance, but it can be done. The trick revolves around explicitly matching leading partial matches to the marker. For example, this POSIX extended regular expression will match everything up to and including the first appearance of substring "M1" in the input, or it will not match if "M1" does not appear at all:

    ^[^M]*(M([^1M][^M]*)?)*M1

Breaking that down,

^[^M]* matches a leading sequence of any number (including zero) of characters other than M
(M([^1M][^M]*)?)* matches any number of subsequences, each starting with an M and
- ([^1M][^M]*)? optionally continuing with
  - [^1M] a character that is neither 1 nor M
  - [^M]* and then any number of characters other than M
  This provides for Ms that are not part of an M1, including runs of Ms of any length, including such runs immediately preceding (but not including) the first M1.
M1 matches itself

That's not too bad for a marker consisting of just two characters, especially when they are different from each other, but it gets nasty very quickly as the number of characters in the marker grows. Fortunately for you, your two problems feature only two-character markers.

That approach can help you get a single s command for each question, depending on to which end of the pattern you apply it. They will be a bit complex, but not outrageously so. I decline to do your homework for you in toto, however, so I leave the details as an exercise.

Ed Morton · Accepted Answer · 2025-03-27 11:06:03Z

2

If you can use sed then you can use awk, the other mandatory POSIX text processing tool, so here's a simple solution using any awk:

$ awk 'match($0,/M1.*M3/) { print substr($0,RSTART+2,RLENGTH-4) }' file
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

$ awk 'sub(/.*M1/,"") && sub(/M3.*/,"")' file
ccc[$cM2ddddM2eeeee

If you really wanted to use sed for some reason, then using any sed that allows \n to mean newline in the regexp and replacement, e.g. GNU sed:

$ sed 's/M1\(.*\)M3.*/\n\1/; s/[^\n]*\n//' file
bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh

$ sed 's/.*M1//; s/M3.*//' file
ccc[$cM2ddddM2eeeee

I expect you can do the same as any of the above in python.

edited Mar 27 at 11:06

answered Mar 27 at 10:58

Ed Morton

209k18 gold badges90 silver badges212 bronze badges

Comments

pjh · Accepted Answer · 2025-03-27 12:23:34Z

2

You can do what you want easily and efficiently with any POSIX-compliant shell, including Bash. No external utilities (sed, perl, python, awk) are required. When run with any non-prehistoric sh this code

string='^[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~'

first_to_last=${string#*M1}
first_to_last=${first_to_last%M3*}

last_to_first=${string##*M1}
last_to_first=${last_to_first%%M3*}

printf '%s\n' "$first_to_last"
printf '%s\n' "$last_to_first"

outputs

bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh
ccc[$cM2ddddM2eeeee

See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for explanations of ${string#*M1} etc.
Also see What is meant by "Now you have two problems"?.

edited Mar 27 at 12:23

answered Mar 27 at 12:13

pjh

8,4232 gold badges20 silver badges21 bronze badges

Comments

Ramanan T · Accepted Answer · 2025-03-28 06:26:23Z

I am posting the solution that I am currently using. Thanks to @markp-fuso and @renaud-pacalet for the ideas input. I am able to solve it.

Python script

import re
text = "^[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~"

# Extract the text between the first M1 and the last 'M3'
result = re.search(r'M1(.*)M3.*$', text).group(1)

# Extract the text between the last M1 and the first 'M3'
result = re.sub(r'.*M1|M3.*','',text)

sed way of extracting is

# Extract the text between the first M1 and the last 'M3'
echo "^[[200~a^[[200~aaa aM1bb bbbM1ccc[\$cM2ddddM2eeeeeM3ffffff f:M3ggggggg M3:hhhhh hhM3:kkkkk~" | sed -E "s/M1(.*)M3.*$/\n\1/;s/.*\n(.*)/\1/"

# Extract the text between the last M1 and the first 'M3'
echo "^[[200~a^[[200~aaa aM1bb bbbM1ccc[\$cM2ddddM2eeeeeM3ffffff f:M3ggggggg M3:hhhhh hhM3:kkkkk~" | sed -E "s/.*M1|M3.*//g"

DuesserBaest · Accepted Answer · 2025-03-27 08:48:06Z

My patterns will not work with sed, but with perl. Since I do not have access to it, i cannot provide example code here, but I do have a python example.

import re

long_pattern=re.compile(r"(?<=M1).+(?=M3)")
short_pattern=re.compile(r"(?<=M1)(?:(?!M1|M3).)+(?=M3)")

s=r"^[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~"

print(
    long_pattern.findall(s),
    short_pattern.findall(s)
)

Long Pattern:

(?<=M1): Assert, that you are to the left of "M1"
.+: and match as much as possible
(?=M3): but ensure the end is followed by "M3"

Short Pattern:

(?<=M1): Assert, that you are to the left of "M1"
(?: ... )+: and match as much as possible
(?!M1|M3).: while making sure you never step over either "M1" or "M3"
(?=M3): but ensure the end is followed by "M3"

Paul Hodges · Accepted Answer · 2025-03-27 21:07:25Z

In python - much easier since it has more robust regex operations, like non-greedy.

import re
txt="[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~"
print( "longest  : " + re.search(r'M1(.*)M3',    txt).group(1) )
print( "shortest : " + re.search(r'.*M1(.*?)M3', txt).group(1) )

Executed:

$: python file.py
longest  : bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh
shortest : ccc[$cM2ddddM2eeeee

Using sed is a little trickier, since it doesn't have as many tools. You say

can contain any special chars

If that's true, I'm not sure a sed regex can help you with absolute certainty without relying on other factors such fixed width fields.

If there is some character you know won't be in the text, such as the newline or null used in other solutions, then those are good.

Otherwise you're going to have to rely on statistics. If you make a specific series of characters as your replacement marker, the length and content of it determines the odds of it happening accidentally in your input stream. A sufficiently unlikely combination is almost as good as an impossibility - you have to decide what constitutes "sufficient".

This still isn't one regex, but in sed it won't matter functionally.

$: cat file

txt='[[200~a^[[200~aaa aM1bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hhM3kkkkk~'
M=" SPECIFIC and UNLIKELY Marker that does NOT have suspicious metacharacters "
printf  "longest  : "
sed -E "s/M1/$M/; s/^.*$M(.*)M3.*$/\\1/" <<< "$txt"

printf "shortest : "
sed -E 's/.*M1//; s/M3.*//;' <<< "$txt"

$: ./file
longest  : bb bbbM1ccc[$cM2ddddM2eeeeeM3ffffff fM3ggggggg M3hhhhh hh
shortest : ccc[$cM2ddddM2eeeee

Alexey Melezhik · Accepted Answer · 2025-04-01 12:57:26Z

1

With Raku/Sparrow you can do like this by zooming in:

# try to find within outer capture
within: "M1" (.*) "M3"
  # then if succeeded
  # zoom in into the capture:
  regexp: "M1" (.*?) "M3"
end:

# dump captured data
code: <<RAKU
!raku
say capture()[0] if capture();
RAKU

It all works because regex are greedy by default and Sparrow zoom in feature allows to start looking from the most outer capture M1 .. M3 group , zooming to the most inner capture group M1 .. M3

edited Apr 1 at 12:57

answered Mar 31 at 16:55

Alexey Melezhik

1,03110 silver badges30 bronze badges

Collectives™ on Stack Overflow

linux sed expression to select text between markers

9 Answers 9

4 Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

4 Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related