0

I have two Bash variables that contain 2 columns of data. I'd like concatenate them to create two larger columns, and then use this outcome to loop in the resulting rows, having each column read in respective temporal variables.

I'll explain what I need with minimal working example. Let's think I have a tmp file with the following sample content:

for i in `seq 1 10`; do echo foo $i; done > tmp 
for i in `seq 1 10`; do echo bar $i; done >> tmp
for i in `seq 1 10`; do echo baz $i; done >> tmp

What I need is effectively equivalent to the following code that relies in external temporary files:

grep foo tmp > file1
grep bar tmp > file2

cat file1 file2 > file_tmp

while read word number
do
  if [ $word = "foo" ]
    then
    echo word $word number $number
  fi  
done < file_tmp


rm file1 file2 file_tmp

My question then is: how can I to achieve this result, i.e. concatenating the two columns and then looping across rows, without having to write out the temporary files file1, file2 and file_tmp?

6
  • you might need paste or somesuch rather than cat if you want to get foo and bar on the same line in file_tmp Commented Aug 16, 2022 at 22:40
  • Why do you grep foo and grep bar but then only test if [ $word = "foo" ]? What's bar got to do with it in that case? Commented Aug 16, 2022 at 23:15
  • @DavidC.Rankin Given the answers and comments I got, I didn't pose the question nicely. The if part of the code was just part of the example I made up to illustrate how my actual problem requires doing something on the second column based on the content of the first. What this code does exactly is silly, I know. Actually I added the if thing altogether at the very end of my edits before publishing the question. I should have discarded it as it distracts from my actual problem. Commented Aug 17, 2022 at 6:49
  • variables that contain 2 columns of data : What exactly does this mean? a variable contains a string. bash also has Arrays (associative and indexed). There is no concept of a "column" in bash. You would need at least to define exactly, what your variables contain. Commented Aug 17, 2022 at 6:55
  • By columns I mean two chunks of characters separated by a space and then a carriage return. This structure (two columns) repeated various times (as many as the number of carriage returns, which is effectively the "number of rows"). Commented Aug 17, 2022 at 7:06

5 Answers 5

3
while
    read -u3 foo1 foo2 &&
    read -u4 bar1 bar2
do
    echo "$foo1 $foo2 - $bar1 $bar2"
done 3< <(grep ^foo tmp) 4< <(grep ^bar tmp)

The code above is a kind of zip function. Note that it doesn't address ensuring that the ordering of the two sequences is correct.


It's not clear why your code in the question creates and then ignores bar lines. If you are doing that, the code is even simpler:

while read word number; do
    echo "word $word number $number"
done < <(grep ^foo tmp)
Sign up to request clarification or add additional context in comments.

8 Comments

Might be a bit easier with grep '^foo\|^bar' tmp?
@DavidC.Rankin I'm assuming a zip function is what's being requested. merging grep wouldn't help with that
Yes, you would need to do them sequentially to separate the columns anyway unless you compared which was found by the combined grep in each line anyway. That was just a through to cut down the number of descriptors used. I'm still not entirely clear what word number is that is being read in the question. Assuming the file format is something like "foo 1234\nfoo 1235\nbar 1234...."
If the question was consistent with the text then a case "$word" in with "foo" and "bar" would be more consistent (or at least both if [ "$word" = ... ]`). Hopefully further comment or clarification by the questioner will make that clear.
Read carefully 2nd paragraph in REDIRECTION chapter of bash's man page: You could use {varname} instead of numbers: while read -ru "$gbar" bar1 bar2 && read -ru "$gfoo" foo1 foo2;.... ;done {gbar}< <(grep ^bar tmp) {gfoo}< <(grep... ! ... (and care about -r switch of read command;)
|
1

I may have misunderstood, but if you want to do this without temp files, perhaps this would work for your use-case:

# Gather the output from the 3 'seq' commands and pipe into AWK
{ 
  for i in $(seq 1 10); do echo foo "$i"; done ;
  for i in $(seq 1 10); do echo bar "$i"; done ;
  for i in $(seq 1 10); do echo baz "$i"; done ; 
} |\
awk '{
  if ($1=="foo" || $1=="bar") {a[NR]=$1; b[NR]=$2}} 
  END{for (i in a) {print "word " a[i] " number " b[i]}
}'

# For the AWK command: if a line contains "foo" or "bar",
# create an array "a" for the word, indexed using the row number ("NR")
# and an array "b" for the number, indexed using the row number ("NR")
# Then print the arrays with the words "word" and "number" and the correct spacing

Result:

word foo number 1
word foo number 2
word foo number 3
word foo number 4
word foo number 5
word foo number 6
word foo number 7
word foo number 8
word foo number 9
word foo number 10
word bar number 1
word bar number 2
word bar number 3
word bar number 4
word bar number 5
word bar number 6
word bar number 7
word bar number 8
word bar number 9
word bar number 10

1 Comment

Thanks @DavidC.Rankin - excellent advice - I've edited my answer to try and explain the commands in more detail. I actually think jhnc's answer is very likely what OP is looking for, but I thought I should post this on the off chance this is actually what OP wants (it's not clear to me what the output should be)
1

you mean like this ??

paste <( jot - 1 9 2 ) <( jot - 2 10 2 )
1   2
3   4
5   6
7   8
9   10

Comments

0

You use awk to achieve this.

awk '{if($1=="foo") {print "word "$1" number "$2}}' file_tmp

3 Comments

How to handle both foo and bar at the same time? Wouldn't something like awk '/^foo|^bar/ {print "word "$1" number "$2}' file_tmp do? (question is a bit unclear on that point)
You can extend the if conditions, awk '{if($1=="foo" || $1=="bar") {print "word "$1" number "$2}}' file_tmp
My question is specifically how to avoid using the intermediate file file_tmp.
0

Splitting then merging standard input in one operation

Of course, this could be used on standard input like output of any command, as well as on a file.

This demonstration use command output directly, without the requirement of temporary file.

First, the bunch of lines:

I've condensed your 1st tmp file into this one line command:

 . <(printf 'printf "%s %%d\n" {1..10};' foo bar baz)

For reducing output on SO, here is a sample of output for 3 lines by word (rest of this post will still use 10 values per word.):

. <(printf 'printf "%s %%d\n" {1..3};' foo bar baz)
foo 1
foo 2
foo 3
bar 1
bar 2
bar 3
baz 1
baz 2
baz 3

You will need a fifo for the split:

mkfifo $HOME/myfifo

Note: this could be done by using unnamed fifo (aka without temporary fifo), but you have to manage openning and closing file descriptor by your script.

tee for splitting, then paste for merging output:

Quick run:

. <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) |
  tee >(grep foo  >$HOME/myfifo ) | grep ba  |
  paste -d $'\1' $HOME/myfifo - - | sed 's/\o1/ and /g'

(Last sed is just for cosmetic) This should produce:

foo 1 and bar 1 and bar 2
foo 2 and bar 3 and bar 4
foo 3 and bar 5 and bar 6
foo 4 and bar 7 and bar 8
foo 5 and bar 9 and bar 10
foo 6 and baz 1 and baz 2
foo 7 and baz 3 and baz 4
foo 8 and baz 5 and baz 6
foo 9 and baz 7 and baz 8
foo 10 and baz 9 and baz 10

With some script in between:

. <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) | (
    tee >(
        while read -r word num;do
            case $word in
                foo ) echo Word: foo num: $num ;;
                * ) ;;
            esac
        done >$HOME/myfifo
      ) |
        while read -r word num;do
            case $word in
                ba* ) ((num%2))&& echo word: $word num: $num ;;
                * ) ;;
            esac
        done
  ) | paste $HOME/myfifo -

Should produce:

Word: foo num: 1        word: bar num: 1
Word: foo num: 2        word: bar num: 3
Word: foo num: 3        word: bar num: 5
Word: foo num: 4        word: bar num: 7
Word: foo num: 5        word: bar num: 9
Word: foo num: 6        word: baz num: 1
Word: foo num: 7        word: baz num: 3
Word: foo num: 8        word: baz num: 5
Word: foo num: 9        word: baz num: 7
Word: foo num: 10       word: baz num: 9

Other syntax, same job:

paste $HOME/myfifo <(
  . <(printf 'printf "%s %%d\n" {1..10};' foo bar baz) | 
  tee >(
    while read -r word num;do
        case $word in
            foo ) echo Word: foo num: $num ;;
            * ) ;;
        esac
    done >$HOME/myfifo
  ) |
    while read -r word num;do
        case $word in
            ba* ) ((num%2))&& echo word: $word num: $num ;;
            * ) ;;
        esac
    done
)

Removing fifo

rm $HOME/myfifo

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.