Executing awk command from python

Question

I am trying to execute the following awk command from a python script

awk 'BEGIN {FS="\t"}; {print $1"\t"$2}' file_a > file_b

For this, I tried to use subprocess as follows:

subprocess.check_output(["awk", 'BEGIN {FS="\t"}; {print $1"\t"$2}',
                         file_a, ">",
                         file_b])

where file_a and file_b are strings pointing to the path of the files.

From this, I am getting the error

awk: cannot open > (No such file or directory)

I'm sure I'm inputing the arguments to subprocess in a wrong way, but I can't figure out what's wrong.

Plumbum might be interesting for you

FlorianK
– FlorianK

2017-05-31 13:06:40 +00:00
Commented May 31, 2017 at 13:06 — FlorianK
– FlorianK, Commented May 31, 2017 at 13:06

obskyr · Accepted Answer · 2017-05-31 13:31:05Z

2

While it may look like it in your shell of choice, >, <, and | are not actually passed as arguments to the program you run. Rather, they're a special part of the shell that the program never gets to see.

Since they're part of the shell, and not part of the OS or program, you have to emulate their effects yourself with the normal facilities the language gives you. In your case, since you're trying to pipe to a file, simply use Python's open() as you would normally. The subprocess API supports arguments to specify stdout, stdin, and stderr, and you can supply any file object for those.

Check it out:

with open(file_b, 'wb') as f:
    subprocess.call(["awk", 'BEGIN {FS="\t"}; {print $1"\t"$2}', file_a], stdout=f)

Since subprocess.check_output redirects output already, it doesn't take the stdout argument. Using subprocess.call avoids this. If you also need the output later in the script, you can instead assign the return value of check_output to a variable, and then save that to file_b.

If you use a lot of shell commands, you might also want to check out Plumbum, which gives you a large set of fairly silly shell-like operator overloads.

edited May 31, 2017 at 13:31

answered May 31, 2017 at 13:07

obskyr

1,5191 gold badge12 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Filipe Aleixo Over a year ago

Very clear explanation, thanks! There's just ", file_a" missing in your code snippet, in the list given to check_output. I tried with check_output but it doesn't allow to set stdout. So I ended up using call.

obskyr Over a year ago

@FilipeAleixo Ah, makes sense that it wouldn't. Its help() documentation has an example of setting stderr, so I figured stdout would be fine too - but I suppose it's not. Updating answer accordingly.

Collectives™ on Stack Overflow

Executing awk command from python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related