docker-compose cannot wait for mysql database

For anybody who is interested, I found a solution to this:

1 - I wrote a python script to connect to the database every second, but with a timeout. I set this timeout to be quite high at 60 seconds, but this seems to work on my computer.

2 - I added the command to wait into my compose file.

It should mean that I can bring up a set of test containers for my website, where I can specify the exact version of Python and MySQL used.

The relevant files are listed below:

Dockerfile:

FROM python:3.6

ENV PYTHONUNBUFFERED 1

RUN mkdir /code
WORKDIR /code

COPY ./ /code/

RUN pip install -r requirements.txt
RUN python manage.py collectstatic --noinput

docker-compose.yml

version: '3'

services:
  mydb:
    container_name: mydb
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_USER=django
      - MYSQL_PASSWORD=secret
      - MYSQL_DATABASE=dbMarksWebsite
    image: mysql:5.7
    ports:
      # Map default mysql port 3306 to 3308 on outside so that I can connect
      # to mysql using workbench localhost with port 3308
      - "3308:3306"
  web:
    container_name: web
    environment:
      - DJANGO_DEBUG=1
      - DOCKER_PASSWORD=secret
      - DOCKER_USER=django
      - DOCKER_DB=dbMarksWebsite
      - DOCKER_HOST=mydb
      - DOCKER_PORT=3306
    build: .
    command: >
      sh -c "python ./bin/wait-for.py mydb 3306 django secret dbMarksWebsite 60  &&
             python manage.py migrate &&
             python manage.py loaddata myprojects_testdata.json &&
             python manage.py runserver 0.0.0.0:8080"
    ports:
      - "8080:8080"
    depends_on:
      - mydb

wait-for.py

'''
I don't like adding this in here, but I cannot get the typical wait-for scripts
to work with MySQL database in docker, so I hve written a python script that
either times out after ? seconds or successfully connects to the database
The input arguments for the script need to be:

    HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT

'''
import sys, os
import time
import pymysql

def readCommandLineArgument():
    '''
    Validate the number of command line input arguments and return the
    input filename
    '''

    # Get arguments
    if len(sys.argv)!=7:
        raise ValueError("You must pass in 6 arguments, HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT")

    # return the arguments as a tuple
    return (sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6]) 

def connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE):
    '''
    for now, just try to connect to the database.
    '''

    con = pymysql.connect(host=HOST, port=PORT, user=USERNAME, password=PASSWORD, database=DATABASE)

    with con:
        cur = con.cursor()
        cur.execute("SELECT VERSION()")

def runDelay():
    '''
    I don't like passing passwords in, but this is only used for a test docker
    delay script
    '''

    # Get the database connection characteristics.
    (HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT) = readCommandLineArgument()

    # Ensure timeout is an integer greater than zero, otherwise use 15 secs a default
    try:
        TIMEOUT = int(TIMEOUT)
        if TIMEOUT <= 0:
            raise("Timeout needs to be > 0")
    except:
        TIMEOUT = 60

    # Ensure port is an integer greater than zero, otherwise use 3306 as default
    try:
        PORT = int(PORT)
        if PORT <= 0:
            raise("Port needs to be > 0")
    except:
        PORT = 3306

    # Try to connect to the database TIMEOUT times
    for i in range(0, TIMEOUT):

        try:
            # Try to connect to db
            connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE)

            # If an error hasn't been raised, then exit
            return True

        except Exception as Ex:
            strErr=Ex.args[0]
            print(Ex.args)
            # Sleep for 1 second
            time.sleep(1)

    # If I get here, assume a timeout has occurred
    raise("Timeout")


if __name__ == "__main__":

    runDelay()

I solved it using the following function in my entrypoint.sh:

function wait_for_db()
{
  while ! ./manage.py sqlflush > /dev/null 2>&1 ;do
    echo "Waiting for the db to be ready."
    sleep 1
  done
}

For testing/development purposes, you could use a version of the MySQL image that has health checks (I believe there's a healthcheck/mysql image), or configure your own (see example here: Docker-compose check if mysql connection is ready).

For production use, you don't want to upgrade the database schema on startup, nor do you want to assume the database is up. Upgrading schema automatically encourages you to not think about what happens when you deploy a bug and need to rollback, and parallel schema upgrades won't work. Longer version: https://pythonspeed.com/articles/schema-migrations-server-startup/

Another option is to use a script to control the startup order, and wrap the web service's command.

In the docker-compose's documentation "wait-for-it" is one of the recommended tools, but other exists.