Interpret as fixed string/literal and not regex using sed

Question

For grep there's a fixed string option, -F (fgrep) to turn off regex interpretation of the search string. Is there a similar facility for sed? I couldn't find anything in the man. A recommendation of another gnu/linux tool would also be fine.

I'm using sed for the find and replace functionality: sed -i "s/abc/def/g"

possible duplicate of Escape a string for sed search pattern — Ingo Karkat
– Ingo Karkat, Commented Jul 24, 2013 at 12:44
It is amazing no one answered your actual question of treating the string like a string instead of a regex. I would give my left arm for that answer right now. I really wish folks answering the question would read the question and answer the question that was asked... — jww
– jww, Commented Jun 22, 2018 at 19:19
Have you tried looking at rust's sd crate? Its -s flag is supposed to mean to treat the find pattern as a non-regex string. Your abc and def patterns don't really show a sample with actual regex-related issues, or else I would have written an sd-based answer for it. Without the -s flag, regex matching applies. — JL Peyret
– JL Peyret, Commented Jun 6, 2024 at 0:11

dannysauer · Accepted Answer · 2020-06-19 05:23:55Z

11

Do you have to use sed? If you're writing a bash script, you can do

#!/bin/bash

pattern='abc'
replace='def'
file=/path/to/file
tmpfile="${TMPDIR:-/tmp}/$( basename "$file" ).$$"

while read -r line
do
  echo "${line//$pattern/$replace}"
done < "$file" > "$tmpfile" && mv "$tmpfile" "$file"

With an older Bourne shell (such as ksh88 or POSIX sh), you may not have that cool ${var/pattern/replace} structure, but you do have ${var#pattern} and ${var%pattern}, which can be used to split the string up and then reassemble it. If you need to do that, you're in for a lot more code - but it's really not too bad.

If you're not in a shell script already, you could pretty easily make the pattern, replace, and filename parameters and just call this. :)

PS: The ${TMPDIR:-/tmp} structure uses $TMPDIR if that's set in your environment, or uses /tmp if the variable isn't set. I like to stick the PID of the current process on the end of the filename in the hopes that it'll be slightly more unique. You should probably use mktemp or similar in the "real world", but this is ok for a quick example, and the mktemp binary isn't always available.

edited Jun 19, 2020 at 5:23

answered Nov 12, 2010 at 0:16

dannysauer

3,8761 gold badge25 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

John Over a year ago

This is a slow solution. echo is called for every line of the file which could fork thousands of echo processes if you are using a large file or multiple files.

dannysauer Over a year ago

echo is typically implemented as a shell builtin, which should fork 0 processes on a modern shell like bash.

dannysauer Over a year ago

You can see the builtins at work by using "strace -e trace=process ./shellscript". Write the above script, and see how many fork calls there are. Using a current Bash on a current Ubuntu, there are 2 fork (actually clone(), which is faster than fork()) calls. One for the subshell where basename is executed, and one for the mv. If you rewrote this using "/bin/echo", on the other hand, then there is actually a new fork for every line. And you could see it taking a lot longer to run. :)

gmatht Over a year ago

Also, to avoid read and echo replacing things like \n and -e you probably want to do something like while read -r line;do printf "%s\n" "${line/$pattern/$replace}";done

dannysauer Over a year ago

There are inaccuracies in that explanation that most of the reasons are built upon ("variable scope is hard" and performance is bad if you use a crappy shell), and the rest is just opinion about "readability" - like putting the same code inside a loop suddenly makes it completely illegible. :)

|

Benjamin Loison · Accepted Answer · 2024-03-03 00:10:07Z

6

Option 1) Escape regexp characters. E.g. sed 's/\$0\.0/0/g' will replace all occurrences of $0.0 with 0.

Option 2) Use perl -p -e in conjunction with quotemeta. E.g. perl -p -e 's/\\./,/gi' will replace all occurrences of . with ,.

You can use option 2 in scripts like this:

SEARCH="C++"
REPLACE="C#"
cat $FILELIST | perl -p -e "s/\\Q$SEARCH\\E/$REPLACE/g" > $NEWLIST

edited Mar 3, 2024 at 0:10

Benjamin Loison

5,7504 gold badges20 silver badges37 bronze badges

answered Nov 9, 2010 at 11:55

Kim Burgaard

3,54821 silver badges11 bronze badges

8 Comments

EoghanM Over a year ago

Has perl got an equivalent of sed's -i? I'm using sed as follows: find -name "myfiles" |xargs sed -i "s/abc/def/g"

EoghanM Over a year ago

I'm seeing your perl example get interpreted as a regex!

EoghanM Over a year ago

$ echo "a.b.c" | perl -p -e 's/./,/g' gives me ",,,,," and not "a,b,c"

Hasturkun Over a year ago

@EoghanM: That's because you need to escape ., otherwise it will match any character.

dannysauer Over a year ago

Perl does have a -i argument which works just like the sed -i arg.

|

Pablo Bianchi · Accepted Answer · 2020-06-17 09:20:00Z

5

If you don't want to escape your string, you can reach your goal in 2 steps:

fgrep the line (getting the line number) you want to replace, and
afterwards use sed for replacing this line.

E.g.

#/bin/sh
PATTERN='foo*[)*abc'    # we need it literal
LINENUMBER="$( fgrep -n "$PATTERN" "$FILE" | cut -d':' -f1 )"

NEWSTRING='my new string'
sed -i "${LINENUMBER}s/.*/$NEWSTRING/" "$FILE"

edited Jun 17, 2020 at 9:20

Pablo Bianchi

2,1681 gold badge29 silver badges36 bronze badges

answered Oct 27, 2016 at 17:30

Bastian Bittorf

5375 silver badges6 bronze badges

1 Comment

papo Over a year ago

great idea. And we can also kick out cut and use standard 'grep': line_num=$(grep -Fn "$PATTERN" "$FILE"); line_num=${line_num%%:*} and it's still POSIX

Benjamin Loison · Accepted Answer · 2024-03-03 00:10:23Z

4

If you're not opposed to Ruby or long lines, you could use this:

alias replace='ruby -e "File.write(ARGV[0], File.read(ARGV[0]).gsub(ARGV[1]) { ARGV[2] })"'

replace test3.txt abc def

This loads the whole file into memory, performs the replacements and saves it back to disk. Should probably not be used for massive files.

edited Mar 3, 2024 at 0:10

Benjamin Loison

5,7504 gold badges20 silver badges37 bronze badges

answered Jun 4, 2015 at 9:26

Hubro

60.1k74 gold badges241 silver badges404 bronze badges

2 Comments

Sorin Over a year ago

gsub interpolates the back-references in the replacement string. You won't be able to replace things like '\2' with '\1'

Hubro Over a year ago

@Sorin True, fixed :)

Benjamin Loison · Accepted Answer · 2024-03-03 00:10:41Z

0

You can do this in two lines of bash code if you're OK with reading the whole file into memory. This is quite flexible -- the pattern and replacement can contain newlines to match across lines if needed. It also preserves any trailing newline or lack thereof, which a simple loop with read does not.

mapfile -d '' < file
printf '%s' "${MAPFILE//"$pat"/"$rep"}" > file

For completeness, if the file can contain null bytes (\0), we need to extend the above, and it becomes

mapfile -d '' < <(cat file; printf '\0')
last=${MAPFILE[-1]}; unset "MAPFILE[-1]"
printf '%s\0' "${MAPFILE[@]//"$pat"/"$rep"}" > file
printf '%s' "${last//"$pat"/"$rep"}" >> file

edited Mar 3, 2024 at 0:10

Benjamin Loison

5,7504 gold badges20 silver badges37 bronze badges

answered Oct 17, 2018 at 19:21

Grisha Levit

8,7182 gold badges42 silver badges55 bronze badges

Comments

Benjamin Loison · Accepted Answer · 2024-03-03 00:11:10Z

0

perl -i.orig -pse  'while (($i = index($_,$s)) >= 0) { substr($_,$i,length($s), $r)}'--\ 
     -s='$_REQUEST['\'old\'']' -r='$_REQUEST['\'new\'']' sample.txt

-i.orig in-place modification with backup.
-p print lines from the input file by default
-s enable rudimentary parsing of command line arguments
-e run this script
index($_,$s) search for the $s string
substr($_,$i,length($s), $r) replace the string
while (($i = index($_,$s)) >= 0) repeat until
-- end of perl parameters
-s='$_REQUEST['\'old\'']', -r='$_REQUEST['\'new\'']' - set $s,$r

You still need to "escape" ' chars but the rest should be straight forward.

Note: this started as an answer to How to pass special character string to sed hence the $_REQUEST['old'] strings, however this question is a bit more appropriately formulated.

edited Mar 3, 2024 at 0:11

Benjamin Loison

5,7504 gold badges20 silver badges37 bronze badges

answered Mar 6, 2020 at 18:15

Sorin

5,4662 gold badges21 silver badges45 bronze badges

Comments

daru · Accepted Answer · 2025-05-05 17:57:33Z

0

I did the "hardwork" going through all special characters in ERE,PERL (I hope)

in /usr/local/bin/ssed

#!/bin/sh

file="$1"

if [ -n "$file" ]; then
  sed "s#[@/\^.$|()[*+?{}\#,&=:~]\|-\|\]#\\\&#g" "$file"
else
  sed "s#[@/\^.$|()[*+?{}\#,&=:~]\|-\|\]#\\\&#g"
fi

Example Usage:

$ grep '^[^#]' mirrorlist | ssed | xargs -d'\n' -I{} sed -i '/{}/ s/#//' mirrorlist.pacnew

In this case I needed to go through all urls in mirrorlist.pacnew and uncomment those which are in mirrorlist uncommented.

answered May 5 at 17:57

daru

511 silver badge5 bronze badges

Comments

JL Peyret · Accepted Answer · 2024-06-06 00:25:45Z

OK, hopefully I understood the question correctly. I think the sd crate for rust might be helpful. It's meant as a replacement for sed and has an -s flag to turn off regexes (for some reason that is not super clear in doc).

Here's a problem pattern with sd '..' as it is a grep for "any two characters.

From "xxx...." I want to replace the ".." with "yy".

`sed` fails

% echo 'xxx....' | sed "s/../yy/"
yyx....

bare `sd` also fails

% echo 'xxx....' | sd '..' 'yy'
yyyyyy.

but `sd -s` works

% echo 'xxx....' | sd -s '..' 'yy'
xxxyyyy

Yes, it will also do it on the contents of a file.

% echo 'xxx....' >> myfile
% echo 'xxx....' >> myfile
% cat myfile
xxx....
xxx....
% sd -s '..' 'yy' myfile  
% cat myfile            
xxxyyyy
xxxyyyy

Brian Clements · Accepted Answer · 2010-11-14 20:09:09Z

-3

You should be using replace instead of sed.

From the man page:

   The replace utility program changes strings in place in files or on the
   standard input.

   Invoke replace in one of the following ways:

       shell> replace from to [from to] ... -- file_name [file_name] ...
       shell> replace from to [from to] ... < file_name

   from represents a string to look for and to represents its replacement.
   There can be one or more pairs of strings.

answered Nov 14, 2010 at 20:09

Brian Clements

3,9451 gold badge28 silver badges26 bronze badges

7 Comments

EoghanM Over a year ago

hmmm, don't have 'replace' installed - it looks to be a MySQL related tool? I have MySql installed; is there a separate install for replace?

Brian Clements Over a year ago

It doesn't come in a separate package, sorry. You could try to find it in the mysql source tree and build it yourself, but that is a lot of hassle. So you have mysql, but not /usr/bin/replace? You could try and reinstall the mysql package (mysql-server).

EoghanM Over a year ago

not in /usr/bin and can't reinstall! I'll accept your answer assuming that it works :)

Serge Stroobandt Over a year ago

Due to the absence of /usr/bin/replace on most systems, this does not appear as a universal answer to me.

Sorin Over a year ago

another major flaw with replace is that it doesn't do backups when doing inplace edits.

|

Collectives™ on Stack Overflow

Interpret as fixed string/literal and not regex using sed

9 Answers 9

9 Comments

8 Comments

1 Comment

2 Comments

Comments

Comments

Comments

`sed` fails

bare `sd` also fails

but `sd -s` works

Comments

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

9 Comments

8 Comments

1 Comment

2 Comments

Comments

Comments

Comments

sed fails

bare sd also fails

but sd -s works

Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

`sed` fails

bare `sd` also fails

but `sd -s` works