1

I have some bash scripts I have been running on Ubuntu 14.04 and 16.04 for well over a year now. Some recent Ubuntu update has broken bash and I cannot figure out how to sort this out.

Example:

#!/bin/bash
INPUT=myinput.txt
OUTPUT=myoutput.txt
ACTION1="0;"

cat $INPUT | while read LINE
do
printf "\t\"~${LINE}\"\t\t$ACTION1\n" >> $OUTPUT
done

my script then loops through the file doing the printf statement as follows but this is the output produced

"~jetmo" 0;
"~spamme" 0;
"~baidu" 0;

example contents of myinput.txt

jetmon
spammen
baidu

Input lines containing a lowercase n character gets stripped out ?? Am I missing some major change in bash that occured, this started happening a week ago and is driving me insane now.

I have tried changing #!/bin/bash to #!/bin/sh with same results.

Also tried this method of looping through the input file and still get the same results.

#!/bin/bash
INPUT=myinput.txt
OUTPUT=myoutput.txt
ACTION1="0;"
while read LINE
do
printf "\t\"~${LINE}\"\t\t$ACTION1\n" >> $OUTPUT
done < $INPUT

Tried suggestion from comments using echo

echo -e -n "\t\"~${LINE}\"\t\t$ACTION1\n" >> $OUTPUT

and now I get this output

-e -n "~jetmo" 0;
-e -n "~spamme" 0;
-e -n "~baidu" 0;

Running hexdump -C myinput.txt gives this output

00000000  6a 65 74 6d 6f 6e 0a 73  70 61 6d 6d 65 6e 0a 62  |jetmon.spammen.b|
00000010  61 69 64 75 0a 0a                                 |aidu..|
00000016

Also changed all variable names to lowercase as suggested by Michael but still getting the same results.

#!/bin/bash
input=myinput.txt
output=myoutput.txt
action1="0;"

cat $input | while read line
do
printf "\t\"~${line}\"\t\t$action1\n" >> $output
done

THE SOLUTION The Mystery is Solved

thanks to everyone, but https://stackoverflow.com/users/96588/l0b0 nailed it on the head. I had an IFS=$'\n' hiding earlier on in my script.

I now have this final, better and safer formatting of printf thanks to the recommendations of Nahuel and Charles and it is working 100% ... cannot thank you all enough.

#!/bin/bash
input=myinput.txt
output=myoutput.txt
action1="0;"
while IFS= read -r LINE
do
printf '\t"~%s"\t\t%s\n' "${LINE}" "$ACTION1" >> "$output"
done < $input1
13
  • Try to check your input file for non-UNIX line endings. It works fine for me on Ubuntu 14.04. You can also replace printf with echo -e -n. Commented Jul 4, 2017 at 13:24
  • 2
    works fine for me on ubuntu 16.04... does this work? printf '\t"~%s"\t\t%s\n' "${LINE}" "$ACTION1"... also a few suggestions - use lowercase for variable names to avoid clash with env variables, see mywiki.wooledge.org/BashFAQ/001 for proper way to read file, and consider using text processing tools like sed instead of bash script Commented Jul 4, 2017 at 13:30
  • Thanks Michael and Sundeep. I tried using the echo -e -n and updated my original question with the results. I actually switched from using all lowercase variable names to uppercase when this started which started improving the problems as nothing was working at all. Even the echo strips off the "n" characters from the input. If I post my entire script section which later uses ed to do in place inserts into marker blocks can you help me get this sorted? I have been running these without issue on 16.04 for a year and on 14.04 too, even inside TravisCI which uses Trusty it does the same thing Commented Jul 4, 2017 at 13:42
  • Show the output of hexdump -C myinput.txt Commented Jul 4, 2017 at 13:47
  • Thanks again Michael, here is the output of that hexdump on this simple test file. 00000000 6a 65 74 6d 6f 6e 0a 73 70 61 6d 6d 65 6e 0a 62 |jetmon.spammen.b| 00000010 61 69 64 75 0a 0a |aidu..| 00000016 updated my original question with the hexdump output. Commented Jul 4, 2017 at 13:54

3 Answers 3

3

This can happen if the internal field separator ($IFS) contains the letter n:

$ IFS=$' \t\nn' read line <<< foon
$ printf '%q\n' "$line"
foo

This is a fairly common mistake. Here is the correct value:

$ printf '%q\n' "$IFS"
$' \t\n'

Example:

$ IFS=$' \t\n' read line <<< foon
$ printf '%q\n' "$line"
foon
Sign up to request clarification or add additional context in comments.

3 Comments

That's true if your n is on the end of the line (with only one variable-name argument passed to read), but that doesn't appear to be the case in the OP's code -- unless I'm missing something?
Yeah, their input file has lines ending with n.
I could KICK myself (aarghhh) .... I had an IFS='\n' hiding in my script, oh my word thank you @l0b0 .... I really thought I was going mad.
1

It's safer to use %s to insert a string with printf for example if it can contain %

printf "\t\"~%s\"\t\t%s\n" "${LINE}" "$ACTION1" >> $OUTPUT

EDIT following comments, with single quotes in first argument because there is no variable expansion

printf '\t"~%s"\t\t%s\n' "${LINE}" "$ACTION1" >> "$OUTPUT"

9 Comments

I'd also suggest quoting "$OUTPUT" -- can get an "invalid redirection" error otherwise.
(And if you switched from double to single quotes for the printf string, you wouldn't need any backslash escaping within them).
Nahuel thanks for your input here too, can you explain why the %s is safer to use?
Charles please can you show me an example using my printf with single quotes ??
@MitchellK %s is safer because first string argument is a format specification is not taken literally ; the characters \ and % have special meaning
|
0

THE SOLUTION The Mystery is Solved

thanks to everyone, but https://stackoverflow.com/users/96588/l0b0 nailed it on the head. I had an IFS=$'\n' hiding earlier on in my script.

I now have this final, better and safer formatting of printf thanks to the recommendations of Nahuel and Charles and it is working 100% ... cannot thank you all enough.

#!/bin/bash
input=myinput.txt
output=myoutput.txt
action1="0;"
while IFS= read -r LINE
do
printf '\t"~%s"\t\t%s\n' "${LINE}" "$ACTION1" >> "$output"
done < $input1

1 Comment

Must really thank you all once again, I always pay my dues. The help from Charles, Lobo and Nahuel has taught me so much about printf, I have modified all my scripts to use this new formatting method and they are working flawlessly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.