Hello everyone hope I find you well!
So I'm trying to setup a new project with Django and a Kafka consumer. I'm using confluent-python package to create my consumer. So I'm facing a very odd situation. When I start my project through my docker entrypoint I run a django management command with the following code:
import simplejson
from confluent_kafka import Consumer
from django.core.management import BaseCommand
from loguru import logger
from apps.<my_service> import KafkaHandler
class Command(BaseCommand):
help = "Start Reporting Kafka Consumer"
def handle(self, *args, **kwargs):
logger.debug("Starting consumer")
consumer = Consumer(
{"bootstrap.servers": "kafka:19091", "group.id": "reporting"}
)
consumer.subscribe(["test_producer"])
kafka_handler = KafkaHandler()
try:
while True:
logger.debug("Consuming")
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
logger.debug(f"Consumer error: {msg.error()}")
continue
event_message = msg.value().decode()
logger.debug(f"Received message: {event_message}")
json_message = simplejson.loads(event_message)
kafka_handler.handle(json_message)
finally:
consumer.close()
After I start my project through command docker-compose up I see that the log "Start consumer" is printed two times and that I have two consumer connections instead of only one so I suspect that this code is being run twice which is very odd. Can someone help me with this situation? Bellow I share my docker-entrypoint
!/bin/bash
set -e
if [[ "${PROJECT_RUN_MODE}" == "production" ]]; then
exec <run production mode>
elif [[ "${PROJECT_RUN_MODE}" == "develop" ]]; then
if [[ $# == 0 ]]; then
echo "Starting dev server..."
exec python "${PROJECT_DIR}/manage.py" start_consumer
else
exec "$@"
fi
else
echo "Unknown run mode (${PROJECT_RUN_MODE}). Aborting..."
exit 1
fi
I tried run the code as a python script with django setup and also as a management command and the problem still persisted.
UPDATE: I also tried to run the command with just one log and it still executes twice bellow I share my Dockerfile and compose.yml
Dockerfile
FROM python:3.11.1-slim as compile-image
# production / develop
ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${PROJECT_DIR}/project_dir"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG DELETE_STATIC=0
ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV DELETE_STATIC ${DELETE_STATIC}
ENV PATH="/opt/venv/bin:$PATH"
WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"
RUN apt-get update && \
apt-get -y install g++ libjpeg-dev zlib1g-dev libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev curl && \
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
python -m venv /opt/venv && \
pip install --upgrade pip setuptools wheel && \
# install requirements
pip install -r "settings/requirements/${DEPLOYMENT_MODE}.txt"
FROM python:3.11.1-slim as run-image
# production / develop
ARG DEPLOYMENT_MODE="production"
ARG BASE_DIR="/opt/project"
ARG PROJECT_DIR="${BASE_DIR}/project_dir"
ARG PHOENIX_PORT="80"
ARG DJANGO_SETTINGS_MODULE="settings.settings"
ARG STOPSIGNAL="SIGTERM"
ENV BASE_DIR ${BASE_DIR}
ENV PROJECT_DIR ${PROJECT_DIR}
ENV PROJECT_PORT ${PROJECT_PORT}
ENV PHOENIX_RUN_MODE ${DEPLOYMENT_MODE}
ENV DJANGO_SETTINGS_MODULE ${DJANGO_SETTINGS_MODULE}
ENV PYTHONUNBUFFERED 1
ENV PATH="/opt/venv/bin:$PATH"
COPY --from=compile-image /opt/venv /opt/venv
RUN apt-get update && \
apt-get -y install supervisor g++ gettext libffi-dev default-mysql-client default-libmysqlclient-dev musl-dev && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
mkdir -p /var/log/supervisor
WORKDIR "${PROJECT_DIR}"
## Add code
COPY . "${PROJECT_DIR}"
# Collect static files directly on the image if the build arg is enabled
RUN chmod +x "${PROJECT_DIR}/docker/docker-entrypoint.sh" && \
cp "${PROJECT_DIR}/docker/supervisord.conf" /etc/supervisor/conf.d/supervisord.conf
# Define entry point
ENTRYPOINT ["${PROJECT_DIR}/docker/docker-entrypoint.sh"]
# Expose server port and declare signal for gracefully stopping the container
EXPOSE ${PROJECT_PORT}
STOPSIGNAL ${STOPSIGNAL}
docker-compose.yml
version: "3"
services:
phoenix:
entrypoint: ["bash", "docker/docker-entrypoint.sh"]
build:
context: ..
dockerfile: docker/Dockerfile
args:
DEPLOYMENT_MODE: develop
stop_signal: SIGINT
environment:
<envs>
ports:
- <ports>
docker-compose.ymlthat it may be executing the script twice