How can I Orchestrate Python Scripts with different virtual envs

Question

How can I use Rocketry (Python) with multiple virtual environments? I try to use Rocketry as Process Orchestrator (because it's easier than Airflow and I don't need to run on different machines).

But I don't know how to run different processes that use different virtualenvs.

(doc of rocketry: https://github.com/Miksus/rocketry)

For now, I saw that I could run some scripts before, but I think that it can be used with CLI only.

from rocketry import Rocketry
from rocketry.conds import daily, secondly

app = Rocketry()

@app.task(daily)
def do_daily(): # task from virtualenv A
    ...

@app.task(daily)
def do_secondly(): # task from virtualenv B
    ...


if __name__ == '__main__':
    app.run()

Could you run 1 rocketry process per virtualenv? That seems like the easiest approach. — Nick ODell
– Nick ODell, Commented Oct 27, 2022 at 21:57
It works, but, i want to see a global visualization... só it's not like if I want a "task scheduler", but an orchestrator. — Felipe Messi
– Felipe Messi, Commented Oct 27, 2022 at 22:08
One approach would be to run a Python subprocess using the subprocess module and subprocess.check_call(). Those are allowed to be in different virtualenvs. — Nick ODell
– Nick ODell, Commented Oct 27, 2022 at 22:10
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. — Community
– Community Bot, Commented Oct 27, 2022 at 22:46
Have you considered other process orchestrators? Maybe something like docker-compose, but for processes: github.com/F1bonacc1/process-compose — fibonacci
– fibonacci, Commented Oct 31, 2022 at 21:09

miksus · Accepted Answer · 2022-11-05 11:46:14Z

Author of Rocketry here. I have thought of at some point making a native virtual env support but at the moment doing that yourself is not difficult either. There are two options:

Use regular function task as a wrapper
Use a command task

Note that I have Windows now thus the location of your Python interpreter is somewhere else than I have if you use a different OS (I have it in env/Scripts/python.exe).

So basically both of these will:

Run the actual Rocketry app in one env/process
Launch a task in a separate thread (or async task)
Call your script using a shell command (this command calls your script with specified virtual env)

Using a regular task

import subprocess

app = Rocketry()

@app.task(daily, execution="thread")
def run_script():
    subprocess.check_call(r'"env/Scripts/python.exe" C:/myscript.py')

Read more from subprocess: https://docs.python.org/3/library/subprocess.html. Note that I used a threaded task as using process launches unnecessary extra child process. There is also an async-subprocess which you may also use and then set execution="async" if you prefer that.

Using a command task

app = Rocketry()

app.task(daily, command='"env/Scripts/python.exe" C:/myscript.py', execution="thread")

if __name__ == '__main__':
    app.run()

There is also an argument cwd (which seems to accept the path as a string at the moment) to change the current working directory.

It also seems the command task uses the default execution thus you may want to specify that as thread here as well (changing the default to be more reasonable for command tasks in the future).

Collectives™ on Stack Overflow

How can I Orchestrate Python Scripts with different virtual envs

1 Answer 1

Using a regular task

Using a command task

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Using a regular task

Using a command task

Comments

Your Answer

Sign up or log in

Post as a guest

Related