Using output piped from sed

Question

I have a sed command that is capturing a single line with sometext. The line in the file it is capturing ends with a linefeed. I am trying to utilize this variable in a pipeline, however, when I attempt to echo, or use it with other commands requiring an input, the result is a blank. Ex: sed '1,1!d' somefile.txt | echo "$1", I know the variable itself is not empty as I can replace echo "$1" with cat $1 and see the correct printout.

edit - I have tried piping to a tr -d and removing the newline. I have confirmed the newline character is gone, yet echos still show blank. Cats do not.

edit 2 - I piped the variable into an if statement ... | if [[ -z $1 ]]; then cat $1; fi it hits the if, is determined to be empty, so runs the cat, which prints a non-empty line to console. If the variable is empty why is cat still printing out information?

What is causing this inconsistency and how can I solve my problem? The ultimate goal is to run the output of one sed, through another to replace specific lines in a target file.

sed '1,1!d' somefile.txt | sed '2,1s/.*/'$1'/' targetfile.txt

Contents of somefile.txt:

these
are
words

Contents of targetfile.txt:

The next line should say these
This line should say these
The previous line should say these

Output of echo after sed:

<empty>

Output of cat after sed:

these

Output of 2nd sed, using input from 1st:

The next line should say these

the previous line should say these

What do you mean create a sample you can test? I have described my process completely. im taking a line from a file that literally contains 3 lines of gibberish while running my own tests. Do you really need me to describe the contents of a gibberish test file? I am simply running the command sed'1,1!d' somefile.txt | echo "$1" , sed'1,1!d' somefile.txt | sed '1,1s/.*/'$1'/' FileToChange.txt" and sed'1,1!d' somefile.txt | cat $1 where the contents of somefile.txt are these\n are\n words\n (each word being its own line of course) As stated, Cat is the only one that prints non blank lines. — Erudaki
– Erudaki, Commented May 18, 2018 at 16:05
I don't mean to sound crass, but everything I just stated is in the question I asked. I don't know how I'm supposed to give you the output after the first sed, or the intermediate output, Because I have already stated, it is blank when not used in cat. Can you please better explain what is unclear? — Erudaki
– Erudaki, Commented May 18, 2018 at 16:15
@Erudaki we do not need you to describe the contents of a gibberish test file, we need you to show us the contents of a test file, gibberish or otherwise. Words are ambiguous - we need a concrete minimal reproducible example with concise, testable sample input and execpected output that helps demonstrate your problem and which we can test a potential solution against to see if it works or not. Also, right now your asking us for help to fix the syntax of what you think is the solution to your problem not how to solve your problem correctly, i.e. it sounds like an XY Problem. See How to Ask if that's not clear. — Ed Morton
– Ed Morton, Commented May 19, 2018 at 0:43
@Erudaki please select your sample data and click the {} editing icon or press ctrl+k... you do not need to use \n... for the given somefile.txt and targetfile.txt, could you please add what is expected output? I'm still not clear what is your requirement (and not sure if you know what $1 means) — Sundeep
– Sundeep, Commented May 19, 2018 at 6:10
examples of question with sample input data, expected output nicely formatted - stackoverflow.com/questions/46870285/… and stackoverflow.com/questions/48920626/… — Sundeep
– Sundeep, Commented May 19, 2018 at 6:14

Ed Morton · Accepted Answer · 2018-05-19 15:08:12Z

You are confused about arguments and input data. Look at this:

$ echo "$1"

$ echo "foo" | if [[ -z $1 ]]; then cat $1; fi
foo

The first argument to my shell, $1 is empty so if [[ -z $1 ]] succeeds. The reason that cat $1 produces output is that you have a fundamental shell programming error in that statement - you aren't quoting your variable, $1. The correct syntax isn't cat $1, it's cat "$1". Look at the difference:

$ echo "foo" | if [[ -z $1 ]]; then cat "$1"; fi
cat: '': No such file or directory

We can simplify the code to make what's happening clearer:

$ echo "foo" | cat $1
foo

$ echo "foo" | cat "$1"
cat: '': No such file or directory

The reason that echo "foo" | cat $1 produces output is that the unquoted $1 is expanded by the shell to nothing before cat is called so that statement is equivalent to just echo "foo" | cat and so cat just copies the input coming in from the pipe to it's output.

On the other hand echo "foo" | cat "$1" generates an error because the shell expands "$1" to the null string before cat is called and so it's then asking cat to open a file named <null> and that of course does not exist, hence the error.

Always quote your shell variables unless you have a specific reason not to and fully understand all of the implications. Read a shell man page and/or google that if you're not sure what those implications are.

wrt another part of your code you have:

sed '1,1!d' somefile.txt | echo "$1"

but, unlike cat, echo neither reads it's input from a pipe nor from a file name passed as an argument. The input to echo is just the list of string arguments you provide it so while echo "foo" | cat will cause cat to read the input stream containing foo and output it, echo "foo" | echo will produce no output because echo isn't designed to read input from a pipe and so it'll just print a null string since you gave it no arguments.

It's not clear what you're really trying to accomplish but I think you might want to replace the 2nd line of targetfile.txt with the first line of somefile.txt. If so that's just:

awk '
    NR==FNR { if (NR==1) new=$0; next }
    FNR==2  { $0 = new }
    { print }
' somefile.txt targetfile.txt

Do not try to use sed to do it or you'll find yourself in escaping/quoting hell because, unlike awk, sed does not understand literal strings, see Is it possible to escape regex metacharacters reliably with sed.

Mark Setchell · Accepted Answer · 2018-05-19 08:46:13Z

You appear to want to extract the first line from file1 and use it to replace the second line in file2.

At the moment, you are extracting that value from the first file with your first sed but sending it to the second sed on its stdin rather than as a parameter ($1).

Your description is confusing so I will use this as file1:

File 1 Line 1
File 1 Line 2
File 1 Line 3

And this as file2:

File 2 Line 1
File 2 Line 2
File 2 Line 3

There are many ways to do this.

Method 1

# Extract line1 from file1
extracted=$(sed '1!d' file1)
# Replace line 2 in file2 with extracted value
sed "2s/.*/$extracted/" file2

Not sure why I feel like a dentist now :-)

If you want to put it all on one line, as some folks like to do:

x=$(sed '1!d' file1) && sed "2s/.*/$x/" file2

Method 2

This one is a bit tricky. It uses the first sed to write a script for the second sed:

sed  's/^/2s|.*|/;s/$/|/;q' file1 | sed -f /dev/stdin file2

If you look at the first sed on its own you will see it is generating a script for the second one:

sed  's/^/2s|.*|/;s/$/|/;q' file1
2s|.*|File 1 Line 1|

If you look at the second sed, you will see it is executing a script passed on its standard input:

sed -f /dev/stdin ...

Method 3

Easier still is awk:

awk 'FNR==NR{if(NR==1)saved=$0;next} FNR==2{$0=saved}1' file1 file2
File 2 Line 1
File 1 Line 1
File 2 Line 3

All you need to notice is that I am passing 2 files to awk and that FNR==NR inside the script means that awk is currently processing the first file, because FNR is the line number in the current file and NR is the total number of lines awk has processed from all files so far. So, when processing the second file, NR is greater than FNR by the number of lines in the first file.

Collectives™ on Stack Overflow

Using output piped from sed

Contents of somefile.txt:

Contents of targetfile.txt:

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Contents of somefile.txt:

Contents of targetfile.txt:

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related