1

looking for some advice with creating a homebrew, primitive version of AWS Datapipeline with extra steps.

I'm wanting to create a Docker image with PHP and MySQL client installed. On start-up I'd like to run a MySQL query on another (remote) server and then parse the output of the query to another file.

  1. Start container with a MySQL query.
  2. Container connects to remote server.
  3. Run MySQL query that was sent in using the connected server.
  4. Query result output to file for further processing by a PHP script.
  5. PHP script reads file output and transforms the data.
  6. Transformed data is then uploaded to a FTP server.

I don't need the assistance with steps 5 and 6 just getting the docker to start and connect to remote server to run a query.

Thanks in advance.

2
  • This isn't the best question for Stackoverflow. Generally you have to provide some more details in order for people to make the effort to help you and generally then only with a very specific point. Especially for a task like this that should be very common. If you provided a dockerfile or setup with logs and most of what you've tried already except for the one tiny piece you're missing (like the connection at step 5) then you may get better help than a general "howto" question like this. See the Stackoverflow FAQ for more info on how to ask better questions. Commented May 25, 2021 at 15:38
  • 1
    Thank you for the advice I will post what I currently have an see if anyone can help correct what I have. Commented May 25, 2021 at 15:42

1 Answer 1

3

if you're using official mysql docker image then

https://hub.docker.com/_/mysql

When a container is started for the first time, a new database with the specified name will be created and initialized with the provided configuration variables. Furthermore, it will execute files with extensions **.sh, .sql and .sql.gz** that are found in /docker-entrypoint-initdb.d. Files will be executed in alphabetical order. You can easily populate your mysql services by mounting a SQL dump into that directory and provide custom images with contributed data. SQL files will be imported by default to the database specified by the MYSQL_DATABASE variable.

Here is example, you can even run custom shell scripts, with shared volume your php container can easily access file generated.

version : '3'
services:
  mysql:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: mysecretpassword
    ports:
      - "3306:3306"
    volumes:
      - "./scripts/add-my-schema.sql:/docker-entrypoint-initdb.d/1.sql"
      - "./scripts/add-data-to-it.sql:/docker-entrypoint-initdb.d/2.sql"
      - "./scripts/generate-some-output.sh:/docker-entrypoint-initdb.d/process.sh"

Here is reference to https://github.com/docker-library/mysql/blob/master/8.0/docker-entrypoint.sh#L56

It contains

# usage: docker_process_init_files [file [file [...]]]
#    ie: docker_process_init_files /always-initdb.d/*
# process initializer files, based on file extensions
docker_process_init_files() {
    # mysql here for backwards compatibility "${mysql[@]}"
    mysql=( docker_process_sql )

    echo
    local f
    for f; do
        case "$f" in
            *.sh)
                # https://github.com/docker-library/postgres/issues/450#issuecomment-393167936
                # https://github.com/docker-library/postgres/pull/452
                if [ -x "$f" ]; then
                    mysql_note "$0: running $f"
                    "$f"
                else
                    mysql_note "$0: sourcing $f"
                    . "$f"
                fi
                ;;
            *.sql)    mysql_note "$0: running $f"; docker_process_sql < "$f"; echo ;;
            *.sql.gz) mysql_note "$0: running $f"; gunzip -c "$f" | docker_process_sql; echo ;;
            *.sql.xz) mysql_note "$0: running $f"; xzcat "$f" | docker_process_sql; echo ;;
            *)        mysql_warn "$0: ignoring $f" ;;
        esac
        echo
    done
}

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.