Run a bash command with variables as a python subprocess

Question

I have this shell command:

$ docker run -it --env-file=.env -e "CONFIG=$(cat /path/to/your/config.json | jq -r tostring)" algolia/docsearch-scraper

And I want to run it as a python subprocess. I thought I'll only need an equivalent of the jq -r tostring, but if I use the config.json as a normal string the " don't get escaped. I also escaped them by using json.load(config.json). With the original jq command the " don't get escaped either and it's just returning the json string.

When I use the json returned as a string in python subprocess i get always a FileNotFoundError on the subprocess line.

@main.command()
def algolia_scrape():
    with open(f"{WORKING_DIR}/conf_dev.json") as conf:
        CONFIG = json.load(conf)
        subprocess.Popen(f'/usr/local/bin/docker -it --env-file={WORKING_DIR}/algolia.env -e "CONFIG={json.dumps(CONFIG)}" algolia/docsearch-scraper')

tripleee · Accepted Answer · 2020-09-24 18:46:17Z

4

You get "file not found" because (without shell=True) you are trying to run a command whose name is /usr/local/bin/docker -it ... when you want to run /usr/local/bin/docker with some arguments. And of course it would be pretty much a nightmare to try to pass the JSON through the shell because you need to escape any shell metacharacters from the string; but just break up the command string into a list of strings, like the shell would.

def algolia_scrape():
    with open(f"{WORKING_DIR}/conf_dev.json") as conf:
        CONFIG = json.load(conf)
    p = subprocess.Popen(['/usr/local/bin/docker', '-it',
        f'--env-file={WORKING_DIR}/algolia.env',
        '-e', f'CONFIG={json.dumps(CONFIG)}',
        'algolia/docsearch-scraper'])

You generally want to save the result of subprocess.Popen() because you will need to wait for the process when it terminates.

answered Sep 24, 2020 at 18:46

tripleee

192k37 gold badges318 silver badges369 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Charles Duffy Over a year ago

Could arguably avoid the load/dump round-trip by putting the literal text of the config file (with any trailing whitespace stripped) on the command line.

jo87casi Over a year ago

Thank you. That was it. But now i get "Unknown shorthand flag 'i' in -it". Somebody has a workaround for that as well?

tripleee Over a year ago

I guess you meant docker run -it?

Collectives™ on Stack Overflow

Run a bash command with variables as a python subprocess

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related