3

Premises

Given a file oneSourceOfTruth.env:

FOO=42
... (many entries)

and a docker-compose.yml:

services:
  my-service:
    dockefile: ./Dockerfile
    env_file: oneSourceOfTruth.env

🏁 Objective

I'd like to have all variables from oneSourceOfTruth.env available in the Dockerfile during the build step via docker compose build as well as in the container during runtime (docker compose up). The variables are static in the sense that they are once set (before the build) and then never change again for a specific build and the containers that spawn from the build.

Unfortunately, it does not work with env_file option as it only passes the env variables to the containers. The env variables are not available during the build (e.g. with a RUN command inside the Dockerfile).

Sidenote: I don't necessarily need to access the env variables directly inside the docker-compose.yml file itself, e.g. we don't have something like the following in our docker-compose.yml file.

args:
   - FOO=$FOO

⭕ Constraint

The oneSourceOfTruth.env file is quite long. For scalability reasons and better code quality, I'd like to regard this file as the "one source of truth" where env variables are declared and set. That is, introducing (removing) an env variable should only imply I have to add (remove) one line in this file alone.

❎ Ways to solve the objective without respecting the constraint.

ARG FOO
ENV FOO $FOO

in your Dockerfile. Do this for every single variable. One user knows what I feel about this approach (see here):

This is not a practical solution whatsoever if you have a big environment file. This makes development tedious and unnecessarily annoying where if you can just read environment variables passed from the compose file. Hard-coding variables is never good, because if you would then want to add more environment variables you have to edit multiple files.

The solution proposed here source oneSourceOfTruth.env && docker-compose build did not work for me (env variables were not populated / were empty).

  • Pass the env varialbes in the environment section in the docker-compose.yml file as described here and here.

Other answers I've checked out:

Avoid XY problem

Maybe this is an XY problem. For the sake of completeness, here is why I'd like to get this problem solved ;)

Inside our Dockerfile, we use the following Rails command to precompile our assets:

RUN DB_ADAPTER=nulldb bundle exec rails assets:precompile

Unfortunately, this will spawn the entire Rails machinery (at least we can avoid connecting to the DB with the DB_ADAPTER=nulldb adapter from here). There existed an option initialize_on_precompile that one could set to false to avoid booting up Rails for this task, see here). However, this option was removed from the Rails codebase, see this commit.

This means we are forced to load env variables also in the precompile task. As stated, with the key env_file in our docker-compose.yml file, the env variables are only available in the containers and not during the build (where we need them for the precompile task). A workaround so far was to use ENV("Foo", nil) everywhere in our ruby on rails code such that the variable default to nil if not specified. This way everything works: during the precompile task, all env variables are nil but we don't need them anyways for the mere task of precompiling our assets. During the production run-time, the env variable will then be available.

But with this approach, we silently ignore the case when we really forget to set an env variable. Then, also during production, the variable will be nil and we only recognize this by the effects it causes. Therefore, our solution so far is really just a workaround and should be fixed. If a variable is not available, it should raise a KeyError during the build step of the project and not just when a user complains that something in our app is not working. This can be achieved by using ENV("Foo") instead of ENV("Foo", nil), but now we get the key error during the precompile task since the env variables are not available in the Dockerfile (where we execute the precompile task). That's exactly the problem ;)

Improve this question
2
  • direnv is an application that sets environmental variables up when you navigate to the directory (by loading up a .envrc file). If your build directory is its own, you could add a .envrc file with all your oneSourceOfTruth.env contents into it in the form of export FOO=42 ... and when you go into your directory to build, all the variables should be accessible in the system environment (not just in docker). You can even split off private variables into its own .envrc-like file and call it from .envrc flie to separate sensitive settings. Maybe this is helpful? direnv.net Commented Mar 4, 2024 at 0:36
  • If people are interested to see how we implemented the solution in a real-world project, see this diff. Commented Mar 28, 2024 at 14:54

2 Answers 2

Reset to default
2

In addition to not matching well with Docker's ENV primitive, the other practical problem with this setup is that the environment gets reset at the end of each RUN command.

If you only need the contents of the file once, then you can use the standard shell . command to read it in within the context of a single RUN command. (Some shells have a similar source command, but it is not part of the shell standard, and I'd avoid it in most cases.)

WORKDIR /app
COPY ./ ./ # includes .env file
RUN . oneSourceOfTruth.env && \
    DB_ADAPTER=nulldb bundle exec rails assets:precompile
# RUN echo $Foo  # environment variables won't be set any more
# CMD echo $Foo  # nor in the eventual container

You can work around the CMD problem by writing a shell script that reads in the environment file and then runs the command it's passed. That script can be the image's ENTRYPOINT.

#!/bin/sh
. /app/oneSourceOfTruth.env
exec "$@"

ENTRYPOINT ["/app/with-environment"]
CMD echo $Foo  # will be set now

If you need the environment to be set on every RUN command, then you can override the Dockerfile SHELL to run this script too. This defines the interpreter that's used for shell-syntax RUN and CMD commands (not JSON-array syntax), and you could inject a wrapper script here too.

SHELL ["/app/with-environment", "/bin/sh", "-c"]
RUN echo $Foo
CMD echo $Foo
# CMD ["/bin/sh", "-c", "echo $Foo"] # will not have the environment

The one caution with this last setup is that a docker run or Compose command: override won't see the Dockerfile SHELL and the environment variables won't be set there. You need the ENTRYPOINT wrapper for that.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks a lot, that helped. Unfortunately, another issue is that while we have the oneSourceOfTruth.env available on the host machine, it's not available in the build context (a github repo). Mounting a volume or a bind mount didn't work (file not found). However, in the github repo we have a dummy docker.env file, so we ended up sourcing that (the assets:precompile task is not even relying on the correct env values). Using a new buildkit of docker seems to allow multiple build contexts, but we get moby-ssh errors with it, so we stick with the old buildkit and the dummy env file.
2

You have the oneSourceOfTruth.env available on the machine, where the Docker build happens, right?

Then you can do something like this:

# ...
WORKDIR /app
COPY oneSourceOfTruth.env /app/oneSourceOfTruth.env
RUN source /app/oneSourceOfTruth.env
RUN DB_ADAPTER=nulldb bundle exec rails assets:precompile
RUN rm -f /app/oneSourceOfTruth.env
# ...

Please bear in mind, that the final Docker image should not contain any secrets. Therefore I'd recommend deleting the oneSourceOfTruth.env from the image again file before finishing the build or do a multi-stage build where you do the build in the first step and copy the build result (without the oneSourceOfTruth.env) to the second build step.

At runtime, pass the env_file in your Docker Compose config like you already did.

Improve this answer

1 Comment

Thanks for your answer which got the main point across. I still accepted David's answer as it went into more detail and also clarified that you have to source the env file in the same RUN command where you want to have the env variables available. It also listed some more caveats to watch out for. Thanks for noting the security aspect here, that's certainly an important point to watch out for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.