File input getting consumed inside the while loop

Question

I'm reading through a lookup file and performing a set of actions for each line in the file. However the while loop only reads the first line in the file and exits. Here's the current code that I have.

sql_from_lkp $lookup

function sql_from_lkp {
    lkp=$1
    while read line; do
        sql_from_columns ${line}
        echo ${line}
    done < ${lkp}
}

function sql_from_columns {
    table_name=$1
    table_column_info_file=${table_name}_columns
    line_count=`cat $table_info_file | wc -l`
    ....
}

By selectively commenting the code, I found that if I comment the line_count line, the while loop goes through every line in the file and works fine. So the input is getting consumed by the cat statement.

I've checked other answers and understood that ssh usually consumes the file inputs inside while loops if -n option is not used. But not sure how to fix this case. Need some help.

As an aside: you should double-quote all your variable references so as not to subject the variable values to word-splitting and globbing; also, Bash won't allow you to call functions before they're defined. — mklement0
– mklement0, Commented Jul 30, 2016 at 1:44
At first blush it looks like you are passing line to sql_from_colums that is expecting a table_name as input? — David C. Rankin
– David C. Rankin, Commented Jul 30, 2016 at 1:47
@David I basically have a table on each line. I refactored the code and removed lot of details to make the scenario easy to appear. — Vinay
– Vinay, Commented Jul 30, 2016 at 2:01
@Vinay : Also suggest the [ shell_check ] tool which will help you identify petty errors pretty easily. — Мона_Сах
– Мона_Сах, Commented Jul 30, 2016 at 4:14

mklement0 · Accepted Answer · 2016-07-30 02:21:06Z

You've mistyped a variable name: $table_info_file should be $table_column_info_file.

If you correct that, your problem will go away.

By referring to a non-extant variable - the mistyped $table_info_file - you're essentially executing cat | wc -l (no filename argument passed to cat) in sql_from_columns(), which makes cat read from stdin.

Therefore - after having read the 1st line in the while loop - the cat command in sql_from_columns() consumes the entire rest of your input (< ${lkp}), which is why the while loop exits after the 1st iteration.

Generally,

You should double-quote all your variable references so as not to subject their values to word-splitting and globbing.
Bash won't allow you to call functions before they're defined, so as presented in your question, your code fundamentally couldn't work.
While the legacy `...` syntax for command substitutions is still supported, it has pitfalls that can be avoided with the modern $(...) syntax.
A more efficient way to count lines is to pass the input file to wc -l via < rather than via cat and a pipeline (wc also accepts filename operands directly, but it then prints the input filename after the counts).
- Incidentally, you probably would have caught your mistyped variable reference more easily had you done that, as Bash would have reported an ambiguous redirect error in the absence of a filename following <.

Here's a reformulation that addresses all the issues:

function sql_from_lkp {
    lkp=$1
    while read line; do
        sql_from_columns "${line}"
        echo "${line}"
    done < "${lkp}"
}

function sql_from_columns {
    table_name=$1
    table_column_info_file=${table_name}_columns
    line_count=$(wc -l < "$table_column_info_file")
    # ...
}

sql_from_lkp "$lookup"

Note that I've only added double quotes where strictly needed to make the command robust; it wouldn't hurt to add them whenever a parameter (variable) is referenced.

Can't believe that I missed that. Takes longer to find this than to write the script itself. Thanks for pointing it out and for the best practices!

Collectives™ on Stack Overflow

File input getting consumed inside the while loop

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related