Building Docker images in GitLab's CI

Last weeks article was about ways to make the docker daemon available inside a container. One of the motivations was GitLab’s CI. While the article discussed three different methods and advocated the docker-machine-based solution, it failed to explain how to actually use the methods in order to set up a GitLab runner and then actually run CI jobs with docker access. This is what this article will illustrate.

This article will show three more-or-less different implementations. The first fails to follow my advice from last week and therefore introduces a security risk. The latter two solutions are rather similar. They follow my advice from last week since they are based on docker-machine.

Targeted docker-builder image

This implementation is based on GitLab runners with the docker socket mounted inside. In order to restrict access from the CI code to the docker daemon, only a specific docker-builder image is allowed, which executes a fixed set of docker commands (e.g. docker build followed by docker push) as the image’s entry point.

Let’s start with the GitLab runner. Add a new runner with the gitlab-runner command line tool to your project (or as a shared runner if this is your GitLab instance). The executor should be docker and we tag it with docker-builder. Open the runner config /etc/gitlab-runner/config.toml and modify the volumes and allowed_images in the runners section in order to match the following example.

[[runners]]
  name = "docker-builder"
  url = "https://gitlab.sauerburger.com/"
  token = " * * * "
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ubuntu"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
    allowed_images = ["gitlab.sauerburger.com:5049/frank/docker-builder:*"]

  [runners.cache]

The added volume exposes the docker socket inside the container. This effectively given any CI code root access to the machine where the runner is executed. The idea is that we only allow a specific image to be executed in the runner. The image should limit access to the docker daemon.

The above runner configuration allows only one image from the newly created docker-builder repository. The image is built from the Dockerfile in the repository. The Dockerfile simply adds a custom entrypoint.

FROM docker

COPY entrypoint.sh /
ENTRYPOINT /entrypoint.sh

The entrypoint then runs the docker commands build, login and push.

#!/bin/sh

cd ${CI_PROJECT_DIR}
docker build -t ${CI_REGISTRY_IMAGE} .
echo "${CI_REGISTRY_PASSWORD}" | docker login -u ${CI_REGISTRY_USER} --password-stdin ${CI_REGISTRY}
docker push ${CI_REGISTRY_IMAGE}

You can see that the docker entrypoint executes a predefined set of docker commands. An example CI job in a different repository can make use of this, for example, with the following .gitlab-ci.yml

build-myapp:
  image: gitlab.sauerburger.com:5049/frank/docker-builder:latest
  tags: 
   - docker-builder
  script: 
   - echo "This is never executed."

Since we set the docker-builder tag, GitLab will schedule this CI job on our runner from above. Since gitlab.sauerburger.com:5049/frank/docker-builder:latest matches the allowed_images configuration, the GitLab runner will run our code. Here, code refers only to the code written in entrypoint.sh. The script body is not executed because the entrypoint of our docker-builder image does not run the command passed to the image.

But wait! We can overwrite the entrypoint in .gitlab-ci.yml.

Yes, indeed. Consider the following CI setup.

image-builder-mallory:
  image:
    name: gitlab.sauerburger.com:5049/frank/docker-builder:latest
    entrypoint: ["/bin/sh", "-c"]
  tags: 
   - docker-image-builder
  script: 
   - docker ps

This CI job overwrites the entrypoint with a simple shell command, such that the script body is executed. If you push this configuration and thus trigger a CI pipeline, you can see all containers currently running on the host.

Running with gitlab-runner 11.0.0 (5396d320)
  on image-builder 8d703628
Using Docker executor with image gitlab.sauerburger.com:5049/frank/docker-builder:latest ...
Pulling docker image gitlab.sauerburger.com:5049/frank/docker-builder:latest ...
Using docker image sha256:060696b0ae865efbe4643dba9b9f3d5fb6cd55507131a909950f2d6cac0f56e3 for gitlab.sauerburger.com:5049/frank/docker-builder:latest
Running on runner-8d703628-project-124-concurrent-0 via host...
Fetching changes...
HEAD is now at c2d679a Add Mallory job
Checking out c2d679a6 as master...
Skipping Git submodules setup
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                  PORTS               NAMES
ba55ca8012ef        060696b0ae86        "/bin/sh -c sh -c ..."   1 second ago        Up Less than a second   2375/tcp            runner-8d703628-project-124-concurrent-0-build-4
Job succeeded

By now, it should be clear that everyone who can add code to a repository has root access to your machine. This issue has been reported at GitLab, but it doesn’t look like there will be a solution in the near future.

To summarize, don’t do this.

docker+machine runner with docker socket

In order to have a more secure setup, I suggest using docker-machine in combination with GitLab runners. Let’s create a new runner with the docker+machine executor and the docker tag. Make sure that MachineDriver is set to "virtualbox" and that you use a unique value for MachineName the runner configuration file /etc/gitlab-runner/config.toml.

So far, our new runner would run docker images inside the VM set up by docker-machine, but we would not have access to docker from the CI code. The simplest method to do this is to mount the docker socket by adding "/var/run/docker.sock:/var/run/docker.sock" to the volumes list as shown in the previous section. The important difference here is that we expose the docker daemon of the VM and not the docker daemon of the host system. This sill bears the security issues, that a CI job could infect the VM itself and then infer with all subsequent jobs run in that VM. The solution is simply to allow only a single CI job per VM before it is deleted. This is what I refer to as disposable VMs. The runner configuration allows you to set this up with the MaxBuilds = 1 option. The relevant section in the configuration file should look as follows.

[[runners]]
  name = "docker-machine"
  url = "https://gitlab.sauerburger.com"
  token = " * * * "
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "ubuntu"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
  [runners.machine]
    IdleCount = 0
    MachineDriver = "virtualbox"
    MachineName = "%s.myrunner.yourdomain"
    OffPeakTimezone = ""
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0
    MaxBuilds = 1

A CI job can then directly access the docker socket and use it to build images and push them to the registry.

image-builder-machine:
  image: docker
  tags: 
   - docker

  script: 
   - docker build -t ${CI_REGISTRY_IMAGE} .
   - echo "${CI_REGISTRY_PASSWORD}" | docker login -u ${CI_REGISTRY_USER} --password-stdin ${CI_REGISTRY}
   - docker push ${CI_REGISTRY_IMAGE}

It doesn’t matter if the CI code accidentally or purpusefully misuses the docker daemon since it’s actions are confined to the VM. Because we have mounted the docker socket and thus gave full access over the daemon to the CI code, the CI job could kill its own container.

docker+machine runner with docker-in-docker

Alternatively, instead of mounting the docker socket, you could also use docker-in-docker. For this, we need to start a privileged container inside the VM. Also, to make sure that every CI job has a fresh and clean VM, we set MaxBuilds to one.

[[runners]]
  name = "docker-machine"
  url = "https://gitlab.sauerburger.com"
  token = " * * * "
  executor = "docker+machine"
  [runners.docker]
    tls_verify = false
    image = "ubuntu"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
  [runners.machine]
    IdleCount = 0
    MachineDriver = "virtualbox"
    MachineName = "%s.myrunner.yourdomain"
    OffPeakTimezone = ""
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0
    MaxBuilds = 1

The configuration for a CI job is slightly more involved and adds a few lines of boilerplate code to the .gitlab-ci.yml. In order to use docker-in-docker in a CI job, we need to run a service and link to it. Consider the following example.

image-builder-dind:
  image: docker 
  tags:
    - docker

  services:
    - name: docker:dind
      alias: docker-in-docker

  variables:
    DOCKER_HOST: tcp://docker-in-docker:2375/

  script:
   - docker build -t ${CI_REGISTRY_IMAGE} .
   - echo "${CI_REGISTRY_PASSWORD}" | docker login -u ${CI_REGISTRY_USER} --password-stdin ${CI_REGISTRY}
   - docker push ${CI_REGISTRY_IMAGE}

When the GitLab runner executes this job inside the VM, it first starts a privileged container: then docker-in-docker container. This container exposes its docker socket on port 2375. The runner then starts a second container in which the script part is executed. We can tell the docker client to connect to the dind-daemon in the other container by setting the DOCKER_HOST variable tcp://docker-in-docker:2375/.

The CI job does not show up in the docker ps because of the separation introduced by docker-in-docker. This means that it is not possible for a CI job to kill its own container by accident.