Using awk to append columns on a file

Question

I have columns of data arriving from standard output (in my case a call to mysql), and I would like to append at each loop the column in a file. How can I do?

Standard output:

 a1
 a2
....
 an

Saving in a file called table.dat:

table.dat:

 a1
 a2
....
 an

Then another output is produced:

Further standard output:

 b1
 b2
....
 bn

Appending to table.dat:

table.dat:

 a1   b1
 a2   b2
.... ....
 an   bn

...and so on. I can use paste, but I need three steps:

 line producing standard output > tmpfile;
 paste prevfile tmpfile > table
 mv table prevfile;

Is there a faster way, maybe by using awk?

This solution: Add a new column to the file produces an empty table.

You can use paste tablefile <(program) to skip one temp file. If you have sponge, you can add | sponge tablefile to do the replacement in-place, otherwise you just need to use a temporary file and rename every iteration. — Kevin
– Kevin, Commented Nov 5, 2013 at 19:14
[mysql call] | paste table - | sponge table is exactly what I needed, thaks! Incredible this "sponge" in the moreutils debian package: I never heard about it! — leonard vertighel
– leonard vertighel, Commented Nov 5, 2013 at 19:50

anubhava · Accepted Answer · 2013-11-05 19:14:22Z

6

You can use paste like this by reading from stdin:

paste <(command1) <(command2)

e.g.

paste <(cat f1) <(cat f2)

instead of:

paste f1 f2

answered Nov 5, 2013 at 19:14

anubhava

790k67 gold badges603 silver badges671 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:33:35Z

2

Just to clarify some details in the case where the two streams given, don't have the same number of elements. Result with paste as proposed by anubhava :

[ ~]$ cat t1.txt 
a1
a2
a3
[ ~]$ cat t2.txt 
b1
b2

[ ~]$ paste t1.txt t2.txt 
a1  b1
a2  b2
a3

Otherwise, with Bash just for fun :

[ ~]$ cat test.sh 
#!/bin/bash

f1=t1.txt
f2=t2.txt

getNumberOfRows(){
    wc -l "$1"|cut -d" " -f1
}

s1=$(getNumberOfRows "$f1")
s2=$(getNumberOfRows "$f2")
[[ $s1 -le $s2 ]] && min="$s1" || min="$s2"

i=1
while read; do
    echo "$REPLY $(sed -n ${i}p $f2)"
   (( i++ ))
   [[ $i -ge $min ]] && break
done < "$f1"

[ ~]$ ./test.sh 
a1 b1
a2 b2
[ ~]$

In this example you could see that we don't display additional lines if a file is larger than the other.

Of course you could change files by command outputs in either paste or with this script ;)

Using paste :

paste <(COMMAND1) <(COMMAND2)

Using the script : see this answer to see how to read a command output in a loop.

edited May 23, 2017 at 12:33

CommunityBot

11 silver badge

answered Nov 5, 2013 at 19:54

Idriss Neumann

3,8682 gold badges26 silver badges33 bronze badges

1 Comment

gniourf_gniourf Over a year ago

For a pure bash solution: while read -u 4 a && read -u 5 b; do echo "$a $b"; done 4< file1 5< file2 is much more efficient than your repeated calls to sed. Enjoy!

Collectives™ on Stack Overflow

Using awk to append columns on a file

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related