1

Hello everyone hope I find you well!

So I'm trying to setup a new project with Django and a Kafka consumer. I'm using confluent-python package to create my consumer. So I'm facing a very odd situation. When I start my project through my docker entrypoint I run a django management command with the following code:

import simplejson
from confluent_kafka import Consumer
from django.core.management import BaseCommand
from loguru import logger
from apps.<my_service> import KafkaHandler


class Command(BaseCommand):
    help = "Start Reporting Kafka Consumer"

    def handle(self, *args, **kwargs):
        logger.debug("Starting consumer")
        consumer = Consumer(
            {"bootstrap.servers": "kafka:19091", "group.id": "reporting"}
        )
        consumer.subscribe(["test_producer"])
        kafka_handler = KafkaHandler()

        try:
            while True:
                logger.debug("Consuming")
                msg = consumer.poll(1.0)

                if msg is None:
                    continue
                if msg.error():
                    logger.debug(f"Consumer error: {msg.error()}")
                    continue

                event_message = msg.value().decode()
                logger.debug(f"Received message: {event_message}")
                json_message = simplejson.loads(event_message)
                kafka_handler.handle(json_message)

        finally:
            consumer.close()

After I start my project through command docker-compose up I see that the log "Start consumer" is printed two times and that I have two consumer connections instead of only one so I suspect that this code is being run twice which is very odd. Can someone help me with this situation? Bellow I share my docker-entrypoint

!/bin/bash

set -e

if [[ "${PROJECT_RUN_MODE}" == "production" ]]; then
    exec <run production mode>
elif [[ "${PROJECT_RUN_MODE}" == "develop" ]]; then
    if [[ $# == 0 ]]; then
        echo "Starting dev server..."
        exec python "${PROJECT_DIR}/manage.py" start_consumer
    else
        exec "$@"
    fi
else
    echo "Unknown run mode (${PROJECT_RUN_MODE}). Aborting..."
    exit 1
fi

I tried run the code as a python script with django setup and also as a management command and the problem still persisted.

UPDATE: I also tried to run the command with just one log and it still executes twice bellow I share my Dockerfile and compose.yml

Dockerfile

FROM python:3.11.1-slim as compile-image

# production / develop
ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${PROJECT_DIR}/project_dir"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG DELETE_STATIC=0

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV DELETE_STATIC ${DELETE_STATIC}
ENV PATH="/opt/venv/bin:$PATH"

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"

RUN apt-get update && \
    apt-get -y install g++ libjpeg-dev zlib1g-dev libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev curl && \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    python -m venv /opt/venv && \
    pip install --upgrade pip setuptools wheel && \
    # install requirements
    pip install -r "settings/requirements/${DEPLOYMENT_MODE}.txt"


FROM python:3.11.1-slim as run-image

# production / develop

ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${BASE_DIR}/project_dir"
ARG PHOENIX_PORT="80"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG STOPSIGNAL="SIGTERM"

ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV PATH="/opt/venv/bin:$PATH"

COPY --from=compile-image /opt/venv /opt/venv
RUN apt-get update && \
    apt-get -y install supervisor g++ gettext libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* && \
    mkdir -p /var/log/supervisor

WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"


# Collect static files directly on the image if the build arg is enabled
RUN chmod +x "${PROJECT_DIR}/docker/docker-entrypoint.sh"  && \
    cp "${PROJECT_DIR}/docker/supervisord.conf" /etc/supervisor/conf.d/supervisord.conf

# Define entry point
ENTRYPOINT ["${PROJECT_DIR}/docker/docker-entrypoint.sh"]

# Expose server port and declare signal for gracefully stopping the container
EXPOSE ${PROJECT_PORT}
STOPSIGNAL ${STOPSIGNAL}

docker-compose.yml

version: "3"

services:
  phoenix:
    entrypoint: ["bash", "docker/docker-entrypoint.sh"]
    build:
      context: ..
      dockerfile: docker/Dockerfile
      args:
        DEPLOYMENT_MODE: develop
    stop_signal: SIGINT
    environment:
    <envs>
    ports:
    - <ports>
9
  • Why do you need Django for this? You can just use argparse for "modes" Commented Jul 10, 2023 at 11:15
  • In my handler I make use of the ORM to process the message I consume Commented Jul 10, 2023 at 11:30
  • Maybe move the consumer variable to a constructor function rather than the handle function Commented Jul 10, 2023 at 11:33
  • Otherwise, make a minimal reproducible example of a command class that only prints... Does it print twice? If so, Kafka isn't the issue Commented Jul 10, 2023 at 11:34
  • This is just an initial POC that's why I haven't moved kafka connection to env variables. You are absolutely correct in that point. I already tried to put just the log and it still prints twice so I think kafka is not the issue. I'm starting to think that it may be because how I run things in docker-compose.yml that it may be executing the script twice Commented Jul 10, 2023 at 11:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.