Task Queue with cached libraries in Django

Question

I am creating a WEB interface for various Python scripts through Django.

Example in calculation.py I would have :

import datetime
def add_time(a, b):
    return = a + b + int(datetime.datetime.now())

Usages :

A user could say "I want to run add from calculation.py with arguments [1, 3]" and that returns him the result when it's ready.
A user could say "I want this to run add from calculation.py with arguments [1, 3] every 10 minutes and check if result is greater than X" and that would action something if it's true.

Most of my script functions are quick but they need to import libraries that takes a lot of time to load.

I am currently doing this directly with my Django service ; it's simple and load the libraries once which allows most of the next calls to be very fast but sometimes a heavy call is made and that is slowing down all my Django application and I seem limited if I want to have CRON scheduling for some scripts. Therefore I am looking at another solution.

I started to look at :

Celery (but it seems not supported anymore on Windows)
Huey and Dramatiq
Django-Q2 (easy to use with Django)

From my understanding nothing will cache the already imported libraries. (I am fine having a "start up bulk import" if needed). Would anyone have a guess on where I should look or what above solution I can tweak ?

Requirement : I need this to run on Windows as some of my libraries are Windows dependant.

To improve performance and schedule tasks in your Django app on Windows, consider using task queue libraries like Huey, Dramatiq, or Django-Q2. Pre-load required libraries at startup to reduce overhead. Implement tasks asynchronously and schedule periodic tasks. This approach offloads heavy tasks, enhances performance, and ensures compatibility with Windows dependencies. — user9416106
– user9416106, Commented Mar 11, 2024 at 10:53

BleuBizarre · Accepted Answer · 2024-03-13 10:36:47Z

0

In the end I used Django-q2 and used the app init to load libraries for qcluster calls

answered Mar 13, 2024 at 10:36

BleuBizarre

3682 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Task Queue with cached libraries in Django

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related