0

I have KEDA + Celery setup working, but there's a timing issue with scaling. I need immediate pod scaling when tasks are queued - essentially maintaining zero pending tasks at all times by spinning up a new pod for each task that can't be immediately processed.

What Happens Now:

  1. Initial state: 1 pod running (minReplicaCount=1), queue=0
  2. Add task 1: Pod picks it up immediately, queue=0, pods=1 ✅
  3. Add task 2: Task goes to queue, queue=1, pods=1 (no scaling yet) ❌
  4. Add task 3: queue=2, pods=1 → KEDA scales to 2 pods
  5. New pod starts: Picks task 2, queue=1, pods=2
  6. Result: Task 3 still pending until another task is added

What I Want:

  1. Add task 1: Pod picks it up immediately, queue=0, pods=1 ✅
  2. Add task 2: Task queued → Immediately scale new pod, new pod picks it up ✅
  3. Add task 3: Task queued → Immediately scale another pod, pod picks it up ✅
  4. Result: Zero tasks pending in queue at any time

Is there a KEDA configuration to achieve "zero queue length" scaling?

# Worker deployment (relevant parts)
containers:
- name: celery-worker
  command:
    - /home/python/.local/bin/celery
    - -A
    - celeryapp.worker.celery  
    - worker
    - --concurrency
    - "1"
    - --prefetch-multiplier
    - "1"
    - --optimization
    - "fair"
    - --queues
    - "celery"


kind: ScaledObject
metadata:
  name: celery-worker-scaler
spec:
  scaleTargetRef:
    kind: Deployment
    name: celery-worker
  pollingInterval: 5
  cooldownPeriod: 120
  maxReplicaCount: 10
  minReplicaCount: 1
  triggers:
    - type: redis
      metadata:
        host: redis-master.namespace.svc
        port: "6379"
        listName: celery
        listLength: "1"
3
  • hi @Shivan, I have the same problem, after some scale the Keda does not scale the tasks, although it completed all queued task, but that is not the ideal solution looking for. If you found any solution, then please inform. Commented Aug 21 at 11:23
  • Hello. If I understand correctly you would like to use activationListLength with value 1 (default to 0). Please let me know if this is solving you issue. Commented Aug 22 at 13:25
  • It didn't work with Redis, RabbitMQ worked fine here. Commented Sep 18 at 6:35

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.