35

I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.

Currently in my cron. directory I have a file called scrapers containing:

* * * * * root /usr/bin/python3.5 /code/scraper.py

This runs the Python script but the script fails, as in the script I use two environment variables.

I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.

SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py

Then I read

In the crontab, before you command, add . $HOME/.profile.

SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py

but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?

Improve this question
9
  • 1
    try adding source ~/.bashrc && command or the file where your env variables are declared and see if it works. Commented Feb 8, 2017 at 14:00
  • @franklinsijo Sorry, I'm not quite sure what I should be doing. The env variables are declared in docker-compose file when setting up the containers. Commented Feb 8, 2017 at 14:12
  • and those variables are the ones to be used by scraper.py? Commented Feb 8, 2017 at 14:13
  • @franklinsijo yes. Commented Feb 8, 2017 at 14:19
  • 1
    AFAIK, env variables has to be set in a file (profile files) to be available even after the session is closed. Or you can use export command to set the variable for that session. Commented Feb 8, 2017 at 15:09

5 Answers 5

Reset to default
11

Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.

The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.

Here's an example setup that I've tested here:

*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py

I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:

export FOO=foovalue
export BAR=barvalue

The /tmp/t10/test.py file had this:

import os

print os.environ["FOO"], os.environ["BAR"]

My cron emails me the output of the scripts it runs. I got an email with this output:

foovalue barvalue
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

setenv && test.py, is equivalent to a system call of test.py in setenv , it will and should definitely work, better approach than mine to put system call of test.py in one file. I like this approach.
Why source it in ~/.profile? Isn't sourcing in the cronjob enough?
You have to source it in ~/.profile if you still want your ~/.profile to work like it did before. If you don't want that, then you don't have to source it in ~/.profile.
I'm trying to implement this since I use zsh, but it's not working. I have two env vars in ~/.env, my .py file sources .env and reads them using os.environ, and this is my cron job: */2 * * * * ~/.env && ~/Documents/Dev/site_monitor/monitor_env/bin/python ~/Documents/Dev/site_monitor/monitor.py. The problem is that I get None for both vars when I print them in the cron job; they still print correctly in VS Code. Originally, the cron job told me there was a permissions error with .env, but I fixed that, so I know it's reading it. I couldn't get the job working with a username
@MarioParra You are missing the . (source) before ~/.env. Without it, you are running .env in a subshell whose variables cease to exist when it exits, and thus they are not available to the rest of the cron job.
|
5

You can set the env variable inline:

* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py

Another way is use honcho that you can pass a file with env variables.

honcho -e /path/to/.env run /code/scraper.py

Improve this answer

2 Comments

Is there a way to have all env values passed through the cron?
Env variables are inherited from the parent process.
4

You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":

crontab

DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"

SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log

my_cool_script.sh

#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command

This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.

Improve this answer

Comments

3

You can specify your two environment variables by this:

* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py

env is a system program that runs a specified program with additional variables:

$ env A=1 B=2 /bin/sh -c 'echo $A$B'  # or just 'sh': would search in $PATH
12
Improve this answer

Comments

1

This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters

set_env_to_process.sh

#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"

export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath

# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
    echo "No command to execute"
else
    echo "Execute commands with its parameters: $@"
    eval $@
fi

usage

/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh set_env_to_process.sh set the correct env for script to run

It could be used as command line, cron, sudo, ssh to setup env

 * * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.