empty value in foreach loop - TCL - unix script

Question

I am getting empty values for the variable $tclItem in the below script in foreach loop even though the list contains values.

Could you please check and see what am i missing?

FTP_USER="xxxxx"
FTP_SERVER="xxxxx"
FTP_PWD="xxxx"
FTP_DROP_DIR="DROP/Archive"
LOGFILE="\tmp\log.txt"

FILES_TO_ARCHIVE="$(cat $LOGFILE | grep '.txt' | awk ' !/Fetching/' | tr -d '\r') "
echo "Files to Archive..."
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
echo $FILES_TO_ARCHIVE
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"


expect <<END 
 spawn sftp $FTP_USER@$FTP_SERVER
 expect "*password: " 
 send "$FTP_PWD\r";
 expect "sftp> "

foreach tclItem {$FILES_TO_ARCHIVE } {
 #puts $tclItem
 send "ls $FTP_DROP_DIR/$tclItem\r"
 expect "sftp> "
};
 send "quit\r"
END

and here is the output that i am receiving.

Files to Archive.....
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
test1.txt test2.txt test3.txt test4.txt
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

spawn sftp [email protected]
abcdef@sftp's password:
sftp> ls DROP/Archive/
sftp> ls DROP/Archive/
sftp> ls DROP/Archive/
sftp> ls DROP/Archive/
sftp> invalid command name "test2.txt"
    while executing
"test2.txt"

In general, it's innately buggy to store lists of filenames in a line-oriented format, since filenames on UNIX are allowed to contain newlines (and spaces, and quotes, and other syntactically-sensitive content) themselves. — Charles Duffy
– Charles Duffy, Commented Mar 26, 2019 at 15:06
Have you considered using a SFTP client built with scripting support, like lftp? Or the Python paramiko library's sftp tooling will similarly be a better tool for the job than expect+sftp. — Charles Duffy
– Charles Duffy, Commented Mar 26, 2019 at 15:06
To be clear about why this behaves as it does -- variable expansions in a heredoc happen once, when the heredoc is generated, which is before expect is invoked. So by the time the foreach tclItem is run by expect, $tclItem was already replaced with an empty value by the shell interpreter at heredoc evaluation time. — Charles Duffy
– Charles Duffy, Commented Mar 26, 2019 at 15:15
Anyhow, using a heredoc to generate code with values that haven't been reviewed by a human is always a bad idea. Someone submits a file with a name that runs an arbitrary command when interpreted by expect, and next thing you know you're running code of their choice -- downloading a rootkit, opening a reverse shell, etc. — Charles Duffy
– Charles Duffy, Commented Mar 26, 2019 at 15:16
before the foreach loop, add puts [list $FILES_TO_ARCHIVE] -- what is the output? — glenn jackman
– glenn jackman, Commented Mar 26, 2019 at 15:40

Charles Duffy · Accepted Answer · 2019-03-26 19:38:45Z

1

Instead of using string substitution to inject data into code (a practice fraught with security concerns), pass your variables from bash to expect through the environment.

# Environment variables should be lowercase except for special/reserved names meaningful
# to the operating system; http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html

ftp_user="xxxxx"
ftp_server="xxxxx"
ftp_pwd="xxxx"
ftp_drop_dir="DROP/Archive"
logfile="/tmp/log.txt"
files_to_archive="$(awk '/[.]txt/ && !/Fetching/' <"$logfile" | tr -d '\r')"

export ftp_user ftp_server ftp_pwd ftp_drop_dir files_to_archive

expect <<'END'
 spawn sftp $env(ftp_user)@$env(ftp_server)
 expect "*password: " 
 send "$env(ftp_pwd)\r";
 expect "sftp> "

 set fileList [split $env(files_to_archive) "\n"]

 foreach tclItem $fileList {
   send "ls $env(ftp_drop_dir)/$tclItem\r"
   expect "sftp> "
 };
 send "quit\r"
END

Note:

Using export causes shell variables to be available as environment variables in subprocesses. In expect, $env(foo) lets one read the environment variable foo.
Using <<'EOF' instead of <<EOF causes bash to pass the code to expect exactly as it is, without changes. (Thus, $tclItem remains $tclItem, instead of being replaced with an empty string before expect even starts).

edited Mar 26, 2019 at 19:38

answered Mar 26, 2019 at 17:29

Charles Duffy

299k43 gold badges442 silver badges498 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Donal Fellows Over a year ago

Even environment variables are imperfect; files and pipes are the best methods, both in terms of security and in terms of the handling of unusual characters. (They're totally the only safe way to move binary data around.) Lots of languages are good at reading and writing files…

Charles Duffy Over a year ago

@DonalFellows, I'd appreciate elucidation of your arguments against use of environment variables. Certainly, they were readable by other users in some older operating systems, but I'm unaware of any OS generally considered to be well-maintained and reasonably secure where that's the case today.

Charles Duffy Over a year ago

@DonalFellows, ...the other thing that strikes me as a reasonable objection is that environment variables can't contain NULs, so they can't be used to represent all possible filename lists without encoding or escaping steps -- but the OP is extracting their names from a newline-delimited file, so the names by definition can't contain newline literals, making the current newline-based splitting adequate to set of possible inputs here. This code should handle names with all possible non-newline characters correctly.

Collectives™ on Stack Overflow

empty value in foreach loop - TCL - unix script

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related