3

I'm trying to understand a specific situation within bash and awk:

I want to use the awk binary operator for string concatenation between two variables (a space) as the variable iterated,$i, within a bash for loop:

$ for i in ' '; do 
  echo "foo bar" | awk '{print $1$i$2}'
done
foofoo barbar

Expected output is: foobar

Question 1

  • What is going on? ANSWER (marked as correct)

Question 2

  • How can I get awk to use string concatenation as in the above bash for loop? ANSWER

Reference

$ $SHELL --version | head -n1
GNU bash, version 4.3.42(4)-release (x86_64-unknown-cygwin)

$ awk --version | head -n1
GNU Awk 4.1.3, API: 1.1 (GNU MPFR 3.1.3, GNU MP 6.1.0)

Full test

$ for i in '+' '-' '*' '/' '%' ' ' ''; do echo "2.0 4.0" | awk '{print $1$i$2}'; done
2.02.0 4.04.0
2.02.0 4.04.0
2.02.0 4.04.0
2.02.0 4.04.0
2.02.0 4.04.0
2.02.0 4.04.0
2.02.0 4.04.0

3 Answers 3

8

It seems to be a little bit tricky one. Actually it prints foo, foo bar and bar. As the value of i is not defined in (it is a variable) it is considered as $0 (I did not know this behavior, but it make sense).

Change the code a little bit as

for i in ' '; do 
  echo "foo bar" | awk '{print $1"<"$i">"$2}'
done

Output:

foo<foo bar>bar

If you want to pass the value of the variable i you can do using -v argument. But $i will not work as the value of i should be a number in $i, so just use simple i.

for i in ' '; do
  echo "foo bar" | awk -v i="$i" '{print $1"<"i">"$2}'
done

Output:

foo< >bar
Sign up to request clarification or add additional context in comments.

17 Comments

I tried -v, but it doesn't work, output of for i in ' '; do echo "foo bar" | awk -v i="$i" '{print $1"<"i">"$2}'; done is foo< >bar, it should perform string concatenation so output is foo<>bar
I tried using my solution to -v and it still doesn't work for i in ' '; do echo "foo bar" | awk -v i="$i" '{print $1"<"'i'">"$2}'; done outputs foo< >bar
@adam if you do not want a space between foo and bar just do not pass the $i to awk! Try awk '{print $1$2}'. The output is foobar. (But I do not understand the importance of $i then... ;) )
i is a regular shell variable, not an environment variable.
Strictly speaking, there is no such thing as an environment variable. What is commonly called such is a shell variable whose value was initialized from the environment, which is just a list of strings of the form name=value. If name happens to be a valid shell identifier, a variable named name is created with the given value. Also, any shell variable x can have its export attribute set, which means the string x=$x (after expansion) will be added to the environment of any child processes.
|
2

I tried making this a comment but there's just too much to say and some of it needs formatting:

@adam you seem to have some fundamental misunderstanding of awk that's making it hard for you to grasp what you're being told. I suspect it comes down to this - awk is not shell. Awk is a completely separate tool/language with it's own scope, variables, functions, etc.

Do NOT try to access the value of shell variables directly within an awk script by using intermediate single quotes (e.g. awk '{print $1'"$i"'$2}') because that will turn the value of the shell variable into part of the awk code before the interpreter reads it and open you up to horrendous errors with cryptic error messages (or worse - insidious bugs with no error messages) given various values of $i.

You say you Cannot get -v to work even without for loop: but then you show it working perfectly twice:

$ echo "foo bar" | awk -v var=" " '{print $1var$2}'
foo bar

In the above case you create an awk variable named var that contains one blank character " ". Then you print $1 (foo) followed by var (" ") followed by $2 (bar) and the output is <foo>< ><bar> exactly as it should be.

In all of your examples you are setting a variable to a single space character, concatenating that with some other values (e.g. -v var=" " then $1var$2) and then for some reason expecting that space character to not be present in the output.

$ echo "foo bar" | awk -v var=" " '{print $1'var'$2}'
foo bar

In the above case you create an awk variable named var that contains one blank character " ". When you write any shell script (awk, sed, grep, whatever) that's enclosed in single quotes:

any_cmd 'abc'

then you are telling cmd to interpret/execute what's inside the sign quotes. You cannot include single quotes in a single quote delimited script - that's shell fundamentals. So when you write:

any_cmd 'abc'def'ghi'

the inner single quotes are actually breaking out of the any_cmd language and back to shell to interpret and the shell attempts to it expand it before any_cmd is called. So if you have:

xyz=17
any_cmd 'abc'$xyz'ghi'

then what any_cmd actually sees to interpret is:

any_cmd 'abc17ghi'

but if you have something in there that the shell can't expand then it gets left as-is so:

and_cmd 'abc'def'ghi'

gets passed to any_cmd as:

any_cmd 'abcdefghi'

So back to your example:

$ echo "foo bar" | awk -v var=" " '{print $1'var'$2}'
foo bar

The var between $1 and $2 will be interpreted by the shell first since the 's around it are taking it out of the awk script and back to shell, but then it's just some text that shell can't expand so the above is passed to awk as-is which makes it:

$ echo "foo bar" | awk -v var=" " '{print $1var$2}'

and in a roundabout way you got back to your first script and again the output is as expected.

The above sounds complicated but it's actually extremely simple:

To concatenate strings in awk, just put them side by side.

To pass the value of a shell variable to awk, use -v, e.g. awk -v awkvar="$shellvar" 'print "foo" awkvar "bar"'.

Rather than trying to learn awk by trial and error, read the book Effective Awk Programming, 4th Edition, by Arnold Robbins first and then play with it.

4 Comments

Thanks for taking the time to compose that. I should have clarified in my post, in the Explore: -v section, I did not expect -v to work. I was just showing the output due to someone else saying it would work. When you say Do NOT try to access the value of shell variables directly within an awk script by using intermediate single quotes... that was actually my question! How can I expand the variable before awk interprets it :) i wouldn't be arbitrary, it would be restricted to the values as listed in the Test section: '+' '-' '*' '/' '%' ' '
I should have used integers or floats in all my examples rather than the string foo bar. I think that is confusing people in misleading to think that I was trying to do a general string concatenation. That was stupid of me.
I updated my answer to include a warning about using it for general string concatenation. Also I updated the example for using -v, this may be more clear what I'm trying to do. echo "2.0 4.0" | awk -v var="+" '{print $1var$2}' output is 2.0+4.0, desired output is 6. Is it possible using -v var="+"? More clear?
Yes it's more clear, no it's not possible, why would you want to do that? It's like if you had a C program tst.c that contained a line 2.0 var 4.0 in it and planned to run it every time as sed 's/var/+/' tst.c > tmp.c && gcc tmp.c using whatever op you want instead of +. Yeah you can physically make it work but I doubt if you'd win any awards for C or shell programming by doing it!
0

Ah, I figured out Question 2.

Initial for loop:

$ for i in ' '; do echo "foo bar" | awk '{print $1$i$2}'; done
foofoo barbar

Explore: quoting

Enclose $i in ":
'{print $1$i$2}' to
'{print $1"$i"$2}'

$ for i in ' '; do echo "foo bar" | awk '{print $1"$i"$2}'; done
foo$ibar

Solution: quoting

Which led me to trying to break up awk with ' so $i can be correctly evaluated:
'{print $1"$i"$2}' to
'{print $1'"$i"'$2}'

$ for i in ' '; do echo "foo bar" | awk '{print $1'"$i"'$2}'; done
foobar

Test solution

Using solution to test all awk binary operators and null:

$ for i in '+' '-' '*' '/' '%' ' ' ''; do 
    echo "2.0 4.0" | awk '{print "Using binary operator \"'"$i"'\" for \""$1"\" and \""$2"\" evaluates to:\t" $1'"$i"'$2}'; 
  done
Using binary operator "+" for "2.0" and "4.0" evaluates to:     6
Using binary operator "-" for "2.0" and "4.0" evaluates to:     -2
Using binary operator "*" for "2.0" and "4.0" evaluates to:     8
Using binary operator "/" for "2.0" and "4.0" evaluates to:     0.5
Using binary operator "%" for "2.0" and "4.0" evaluates to:     2
Using binary operator " " for "2.0" and "4.0" evaluates to:     2.04.0
Using binary operator "" for "2.0" and "4.0" evaluates to:      2.04.0

Note: '' is not a binary operator obviously and is just a check.

Success!

Warning: as pointed out by others, this solution is not for general string concatenation. The shell variable i should only be awk binary operators. Setting i to any other variable will probably cause issues.

Explore: -v

Using -v does not appear to allow binary operation:

$ echo "2.0 4.0" | awk -v var="+" '{print $1var$2}'
2.0+4.0

Output: 2.0+4.0
Desired output: 6

5 Comments

Note that the shell requires you to use $ to get the value of a variable. awk does not (it's like C in that regard). In awk $ is like an operator to get the value of the field given by the variable: echo a b c | awk '{print $2; x=2; print x; print $x}' will output b, 2, b
You want awk -v var=":" '{print $1 var $2}'
@glennjackman I understand $<number> is used to reference the current record, delimited according to the variable NS. Obviously if I just wanted a solution I would do echo "foo bar" | awk -v var="" '{print $1'var'$2}' to get output as foobar, or just not use -v at all. Also, I do not want a separator as your suggestion provides, :. What I'm trying to do is have the binary operator trigger via <space> to concatenate two variables. I should be able to iterate over all binary operators as shown in Test sect. If you have a solution to do this using -v I would love to hear it :)
As has been mentioned elsewhere, to concatenate two strings, you want awk '{print $1 $2}'
Sorry, it's probably my fault. :( I should have used integers in all my examples rather than the string foo bar I think that is confusing a lot of people. I want to either pass via variable any of the binary operators (e.g. +, <space> , *, etc) into the print command between the two vars. I solved it with for loop, but I don't think it is possible with -v in a for loop. Thanks anyway for your help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.