How to assign variable from a shell command to python script

Question

I am trying to run a batch process using array in slurm. I only know shell command to extract variable from array (text files), but failed to assign it as Python variable.

I have to assign a variable to a Python slurm script. I used a shell command to extract values from the array. but facing errors while assigning it to the variable. I used subprocess, os.system and os.popen. or is there any way to extract values from text file to be used as a Python variable?

start_date = os.system('$(cat startdate.txt | sed -n ${SLURM_ARRAY_TASK_ID}p)')

start_date = subprocess.check_output("$(cat startdate.txt | sed -n ${SLURM_ARRAY_TASK_ID}p)", shell=True)

start_date = os.popen('$(cat startdate.txt | sed -n ${SLURM_ARRAY_TASK_ID}p)').read()


start_date = '07-24-2004'

Barmar · Accepted Answer · 2019-10-14 21:57:03Z

1

Don't use $(...). That will execute the command, and then try to execute the output of the command. You want the output to be sent back to python, not re-executed by the shell.

start_date = subprocess.check_output("cat startdate.txt | sed -n ${SLURM_ARRAY_TASK_ID}p", shell=True)

answered Oct 14, 2019 at 21:57

Barmar

789k57 gold badges555 silver badges669 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Malcolm · Accepted Answer · 2019-10-25 17:11:02Z

0

Barmar is correct, the $(...) part is why you are not getting what you want, but the real question is why when you are using python would you want to use cat and sed as well. Just open the file and pull out the information you want

import os
with open("startdate.txt", "r") as fh:
    lines = fh.readlines()
start_date = lines[os.environ['SLURM_ARRAY_TASK_ID']].strip()

the .strip() part gets rid of the newline character.

edited Oct 25, 2019 at 17:11

answered Oct 14, 2019 at 22:11

Malcolm

8815 silver badges14 bronze badges

3 Comments

Arun Bawa Over a year ago

Thanks Malcolm. but I got this error: NameError: name 'SLURM_ARRAY_TASK_ID' is not defined. Is 'SLURM_ARRAY_TASK_ID' should be the column name in text file?

ciaron Over a year ago

SLURM_ARRAY_TASK_ID is only set if you submit the SLURM job as an array (i.e. with --array)

Malcolm Over a year ago

Sorry, for the delay. I missed that in your example ${SLURM_ARRAY_TASK_ID} is actually getting expanded from the environment variables in the execution of your command string. To get that in your python script you need to pull it from os.environ. I will edit my code above to show how to do that as well

Collectives™ on Stack Overflow

How to assign variable from a shell command to python script

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related