2

I'm trying to list some ftp directories. I can't work out how to make bash execute a command that contains pipes correctly.

Here's my script:

#/bin/sh

declare -a dirs=("/dir1" "/dir2")           # ... and lots more
for d in "${dirs[@]}"
do
    cmd='echo "ls /mydir/'"$d"'/*.tar*" | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1'
    $cmd
done

This just outputs:

"ls /mydir/dir1/*.tar*" | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1
"ls /mydir/dir2/*.tar*" | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1

How can I make bash execute the whole string including the echo? I also need to be able to parse the output of the command.

Improve this question
8
  • Try with using eval $cmd. Commented Sep 4, 2014 at 10:19
  • 7
    Don't do that. mywiki.wooledge.org/BashFAQ/050 Commented Sep 4, 2014 at 10:23
  • 2
    As an aside, #!/bin/sh is wrong when your script uses arrays or other Bash-only features. You want #!/bin/bash (or whatever the correct path is on your platform). Commented Sep 4, 2014 at 10:26
  • @tripleee eval works for now and this is a quick-and-dirty script so it's good enough. thanks for the shebang pointer. Commented Sep 4, 2014 at 10:27
  • 2
    Why are you storing the command in a string in the first place? It just complicates your code for no good reason. Commented Sep 4, 2014 at 12:52

3 Answers 3

Reset to default
3

I don't think that you need to be using the -b switch at all. It should be sufficient to specify the commands that you would like to execute as a string:

#/bin/bash

dirs=("/dir1" "/dir2")
for d in "${dirs[@]}"
do
    printf -v d_str '%q' "$d"
    sftp -i ~/mykey [email protected] "ls /mydir/$d_str/*.tar*" 2>&1 | tail -n1       
done

As suggested in the comments (thanks @Charles), I've used printf with the %q format specifier to protect against characters in the directory name that may be interpreted by the shell.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

You should probably quote the wildcards so as to not get a surprise when a glob gets expanded locally.
This doesn't work. The '-b' option indicates a batch file containing commands to execute on the remote server. '-b -' lets it take the command from stdin. Therefore I echo the command to be executed remotely and pipe it into sftp. It's a kludge but that's how sftp works (at least on OSX).
@TomFenech No, it still doesn't work. I've solved this anyway. But you really do have to "echo 'ls /the/path' | sftp -b - ...". The only alternative I'm aware of is putting 'ls /the/path' in 'myfile' and calling sftp with 'sftp -b myfile ...'. Thanks anyway.
@jbrown looking at the manual I don't think that you need to be using the -b switch at all. I have edited my answer.
printf -v d_str '%q' "$d";` then, sftp ... "ls /mydir/$d_str/*.tar*" if you want to work correctly with directory names containing literal doublequote symbols, literal $() strings, literal backticks, or the like.
|
1

First you need to use /bin/bash as shebang to use BASH arrays.

Then remove echo and use command substitution to capture the output:

#/bin/bash

declare -a dirs=("/dir1" "/dir2")           # ... and lots more
for d in "${dirs[@]}"
do
    output=$(ls /mydir/"$d"/*.tar* | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1)
    echo "$output"
done

I will however advise you not use ls's output in sftp command. You can replace that with:

output=$(echo "/mydir/$d/"*.tar* | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1)
Improve this answer

4 Comments

I'm not using 'ls's output with sftp. I'm echoing an ls command to sftp. The exact command I'm trying to script is: echo "ls /mydir/dir1/*.tar*" | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1
Yes I understand that and if you see my first solution I kept your ls command there. However if your file names contain space/new lines then it can create problems.
This doesn't work. It's trying to 'ls' on my machine and then pipe the output to sftp, instead of sending the ls command to sftp to be executed remotely. Anyway, eval works so I've got a solution. Thanks anyway.
@jbrown, eval "works"; mind the scare quotes. If someone wanted to generate a filename that would cause your eval-based "solution" to do something evil/nasty, that's not so hard to do. Mind you, it's not just intentional attacks you need to worry about -- I've seen TB of backups deleted on account of sloppy scripting practices and a buffer overflow bug putting " * " into a filename that was expected to only ever match ^[a-z0-9]+$.
1

Don't store the command in a string; just use it directly.

#/bin/bash

declare -a dirs=("/dir1" "/dir2")           # ... and lots more
for d in "${dirs[@]}"
do
  echo "ls /mydir/$d/*.tar*" | sftp -b - -i ~/mykey [email protected] 2>&1 | tail -n1
done

Usually, people store the command in a string so they can both execute it and log it, as a misguided form of factoring. (I'm of the opinion that it's not worth the trouble required to do correctly.)

Note that sftp reads from standard input by default, so you can just use

echo "ls ..." | sftp -i ~/mykey [email protected] 2>&1 | tail -n1

You can also use a here document instead of a pipeline.

sftp -i ~/mykey [email protected] 2>&1 <<EOF | tail -n1
ls /mydir/$d/*.tar.*
EOF
Improve this answer

2 Comments

Actually I think it's even simpler than that. I don't think that the -b is necessary at all.
I just read the sftp docs; I agree, the -b is only needed if you want to read from a file other than standard input.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.