awk and shell variables | using a changing variable

Question

Supose I have a loop (bash shell), following sort of structure:

for iter in `seq 1 to 5`
  do
    <bunch of stuff that generates a file called test.dat>
    awk -v var="$iter" '{print $1 " " var}' test.dat > mod_test.dat
done

What I can't quite figure out is why awk is not writing out the current value of the $iter variable.

For example, test.dat might look like (say)

abcd
efgh
ijkl
mnop

I'm looking to end up with

abcd 1
efgh 2
ijkl 3
mnop 4

[Basically, I simply want to put the loop counter at the end of each line.]

But, my script so doesn't work, since the output file (mod_test.dat) is simply

 abcd 5
 efgh 5
 ijkl 5
 mnop 5

In other words, it is only appending the last value of the counter to each line. I know the counter is working (echo $iter in the loop shows it is incrementing).

What is the obvious thing I'm doing wrong? I've tried every trick I know to set the awk variable (var=$iter, var="$iter", var="$iter"....), without success. I'm an awk semi-newb, so apologies if this is at a level even below trivial.

Many thanks in advance.

Could you please be more clear here what you are trying to achieve, because if you simple want to print increasing numbers you can do it in awk itself. Also on another note: your seq command should be seq 1 5. — RavinderSingh13
– RavinderSingh13, Commented Apr 1, 2021 at 15:12
Also, test.dat > mod_test.dat overwrites the target file with the last iteration output of awk (which also isn't probably what you're after). Use >> to append to the target file. — James Brown
– James Brown, Commented Apr 1, 2021 at 15:13
Yup - typo entering the seq code. Thanks for point it out. I want to append the counter to the end of each line of the file. I can't figure out why the code at present is only adding the last value of the counter. — Johnny Canuck
– Johnny Canuck, Commented Apr 1, 2021 at 15:14
why the code: First iteration: var=1 and awk processes every record in the file and outputs to target file. Second iteration, var=2 and awk processes every record in the file and outputs to target file. And so on. Lose the bash loop and use only awk '{print $0,NR}' file is the quickest fix. — James Brown
– James Brown, Commented Apr 1, 2021 at 15:16
with respect to Use grep to find the line in the listing containing value I need. Use awk to pull the value from that line, - you never need grep when you're using awk. grep 'foo' | awk '{bar}' = awk '/foo/{bar}'. — Ed Morton
– Ed Morton, Commented Apr 1, 2021 at 19:19

glenn jackman · Accepted Answer · 2021-04-01 15:18:35Z

4

You're processing the file 5 separate times. What you see is just the 5th time you're processing the file, each prior iteration gets overwritten by the next.

awk can handle this by itself:

most simply: awk '{print $0, NR}' file

or with a counter: awk '{print $0, ++n}' file -- you might want to do this if you don't want to count empty lines:

$ printf "%s\n" one two "" three "" four five | awk 'NF > 0 {$(NF+1) = ++n} 1'
one 1
two 2

three 3

four 4
five 5

answered Apr 1, 2021 at 15:18

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ed Morton · Accepted Answer · 2021-04-01 19:23:47Z

3

As already pointed out you're overwriting mod_test.dat on every iteration of the loop and you're also printing every line of test.dat every time you call awk.

It sounds like you're trying to do something like this:

for iter in {1..5}; do
    <bunch of stuff that generates a file called test.dat>
    awk -v var="$iter" '/look for something/{print $1, var; exit}' test.dat
done > mod_test.dat

answered Apr 1, 2021 at 19:23

Ed Morton

209k18 gold badges90 silver badges212 bronze badges

3 Comments

Johnny Canuck Over a year ago

Sure, but I've not been able to 'look for something' without grepping for it first -- that gives me the line, and then awk to pull the 'data' (value) from that line. If there is a way to do it all in awk, sign me up. Basically, I have a 300-400 line output file from the simulation, and need 1 number from the whole thing -- its trivial with grep to find the line it sits on (its preceding by a unique character string). It what happens next that led me to using awk.

Ed Morton Over a year ago

awk '/look for something/{print}' replaces grep 'look for something' | awk '{print}'. It's every bit as trivial with awk as it is in grep but we can't tell you how to do whatever it is you want to do without seeing some sample input and you telling us what it is you want to extract from that input. Please update your question to show that minimal reproducible example and then we can tell you exactly how to write the code.

Ed Morton Over a year ago

Actually it would be best if you just accept one of the answers you got to the question that you originally asked and then ask a followup question about how to just use awk instead of grep + awk.

Collectives™ on Stack Overflow

awk and shell variables | using a changing variable

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related