Pass environment variables from docker-compose to container at build stage

Question

I am trying to create configuration for several environments with a single Dockerfile, several docker-compose files and several envoronment_variables files. I need it to use a different file with python dependencies for each environment.

Let's say, we create a web service in two environments: development and production. To achieve this, I create the following file structure:

docker-compose-dev.yml
docker-compose-prod.yml
envs/
  dev.env
  prod.env
web/
  Dockerfile
  requirements_dev.txt 
  requirements_prod.txt

The objective is to instantiate the proper name of a requirements_*.txt file in the Dockerfile during the container's build process. According to the documentation, I tried this naïve approach, which doesn't seem to work:

Define the name as an environment variable:
- envs/dev.env: REQUIREMENTS=requirements_dev.txt
- envs/prod.env: REQUIREMENTS=requirements_prod.txt

Use this environment variable in the Dockerfile:

FROM python:3.5
ENV PYTHONUNBUFFERED 1
ENV APP_ROOT /usr/src/app
...
COPY $REQUIREMENTS $APP_ROOT/
RUN pip install -r $APP_ROOT/$REQUIREMENTS

Import the corresponding definition in a docker-compose configuration:

version: '2'

services:
    web:
        build: ./web
        env_file:
            - envs/dev.env   # in docker-compose-dev.yml
            - envs/prod.env   # in docker-compose-prod.yml
        ...

When I run docker-compose up --build docker-compose-dev.yml, the $REQUIREMENTS variable is not defined at the build process.

In the docker-compose documentation there is a note on this:

Note: If your service specifies a build option, variables defined in environment files will not be automatically visible during the build. Use the args sub-option of build to define build-time environment variables.

But to use the ARG option in the Dockerfile, like ARG REQUIREMENTS, one needs a corresponding args option in docker-compose, like this:

services:
    web:
        build:
            context: ./web
            args:
                - REQUIREMENTS

Nevertheless, with this configuration my tests also failed, because, according to the documentation, the value of the REQUIREMENTS variable in Dockerfile is taken from

the environment where Compose is running

Therefore, it takes value from my shell, but not from envs/dev.env.

So my question is:

Is it possible to pass environment variable from env_file in docker-compose to Dockerfile at build time without setting variables locally in my shell?

I just ran into this myself. Trying to use env_file and also build: args: FOO: ${SOME_VAR:-foo} and it seems SOME_VAR is not loaded from the env_file.. Perhaps this is a bug where env_file interpolation happens after or does not apply to build arg resolution? — ColinM
– ColinM, Commented Jan 19, 2017 at 20:37
I've read in some other question a statement that the Dockerfile is designed to be self-contained, and to not depend on the external environment. But, on the other hand, it contradicts the purpose of args and ARG. So it's not clear whether it's a bug or a feature. — BartoNaz
– BartoNaz, Commented Jan 19, 2017 at 22:58

and1er · Accepted Answer · 2019-05-06 12:11:42Z

63

Your docker-compose.yml should look like this:

version: '2'
services:
  web:
    build:
      context: ./web
      args:
        REQUIREMENTS: "requirements_dev.txt"

Your Dockerfile should define the build-argument using ARG like this:

FROM python:3.5
ENV PYTHONUNBUFFERED 1
ENV APP_ROOT /usr/src/app
ARG REQUIREMENTS
...
COPY $REQUIREMENTS $APP_ROOT/
RUN pip install -r $APP_ROOT/$REQUIREMENTS

I validated this and created a minified functional demo at Github:

https://github.com/jannikweichert/stackoverflow-41747843

edited May 6, 2019 at 12:11

and1er

1,15112 silver badges22 bronze badges

answered Jan 22, 2017 at 14:58

Jannik Weichert

1,67316 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BartoNaz Over a year ago

Thanks, this works indeed! Though, one needs to hardcode the variables in the docker-compose.yml instead of reading them from an *.env file, but this is much better than nothing.

Jannik Weichert Over a year ago

Great! the good thing is, that as your Dockerfile is a specification with explicit information on necessary input variables, your docker-compose is an explicit specification of your container setup. No additional files need to be checked by a developer to see what's going on

David Maze · Accepted Answer · 2024-10-18 13:05:48Z

Running a container with a build: block happens in two separate stages: first Compose builds the image, and then it runs a container based on that image. Nothing outside the build: block affects the build step, and more specifically, environment variables set via env_file: only affect the running container and aren't visible during the build.

Similarly, Compose has two different notions of "environment". As the Compose file itself is processed, before anything is started, Compose takes into consideration environment variables visible to the docker-compose command, any docker-compose --env-file options, and the implicit .env file. When containers are run, each container's environment: and env_file: set the environment for that container specifically.

For a variable to be visible to the build process, you need several steps:

The variable itself must be declared as a Dockerfile ARG allowing it to be passed in from outside. For example,
```
FROM python:3.5
WORKDIR /usr/src/app
ARG REQUIREMENTS
COPY $REQUIREMENTS ./
RUN pip install -r $REQUIREMENTS
```
ARG values are visible in places like COPY commands and RUN instructions, but not to the eventual container CMD. Note that they are embedded in the image and tools may be able to extract them, which means you should avoid passing credentials this way.
In the Compose file, declare build: { args: } either setting or passing through the variable. The syntax in the question is correct to pass it through from Compose's environment; repeating that
```
services:
  web:
    build:
      context: ./web
      args:
        - REQUIREMENTS
```
Make the value visible in Compose's environment when you run it. You can do this with a normal environment variable on the host
```
REQUIREMENTS=requirements_dev.txt docker-compose build
```
but if you want to use the existing environment file, there's a Compose CLI option for it
```
docker-compose --env-file envs/dev.env build
```

If this file only contains build-time settings, you don't necessarily need to include it in the Compose env_file:, and you may be able to delete that line or block entirely. Since the file will be loaded into Compose's outer environment, you can also use environment: to selectively pass through values or map them to different names if needed.

Collectives™ on Stack Overflow

Pass environment variables from docker-compose to container at build stage

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related