Picture a typical GitLab CI/CD deployment pipeline of a web app. The actual technical implementation does not matter. The deployment is most likely configured through variables. For example, the app receives the following set of variables:

  • The hostname and port of an external database,
  • The credentials of the external database,
  • The private key of the apps TLS certificate, and
  • Application-specific configuration settings (colors, images, URL paths, etc).

What should be a secret GitLab variable and what should be hard-coded in the repository (e.g. .gitlab-ci.yaml or a config file)?

GitLab secrets have the obvious benefit of being secret. Hard-coded variables have the appealing property of being tracked and versioned by Git. What’s the best balance between these two options?

Requirements

In order to answer this, let’s establish requirements. We want to be able to

  1. Redeploy an old (esp. the previous) deployment pipeline to revert back to an old version, and
  2. Secrets must not be committed to the Git repository.

The first requirement is important if a deployment does not work as intended. In case of any issue, it should be safe and easy to revert back to the previous version.

Rules

Based on these requirements we can derive the following best practice rules:

  1. Prefer hard-coded values in the repository unless another role forbids it.
  2. Any secret must be stored in a GitLab secret variable.
  3. Any variable referring to an external system whose value might change independently should be a secret variable.

Justification

The first rule takes advantage of Git. When we change a hard-coded variable in the Git repository, we have the full history of that change. Rerunning an old pipeline will also reset the variable to its former glory. In contrast to that, if we modify a GitLab variable, we lose any trace of its former value. Rerunning an old pipeline will use the new set of variables, potentially breaking the old pipeline. This could have catastrophic consequences if a new deployment fails and the previous deployment cannot be restored.

The second rule should be obvious and doesn’t need more justification. Never commit a secret.

So what about the third rule? Why sacrifice the benefits of Git in the case of external systems? The reason for this rule is also rooted in our first requirement. Assume the example from above. The hostname and port of the external database are stored as a secret. The database might be migrated to a new server. So in the course of the migration, we change the variable and redeploy the application. So far, so good.

Now, let’s further assume we discover an issue with the app, unrelated to the new database, and we want to redeploy the previous version. This is not a problem. The previous deployment will also reuse the new, updated database endpoint.

If, on the other hand, we had hard-coded the external database, we could not redeploy an old release from before the migration.

Example

To conclude with the example from above, the variables should be implemented in the following ways:

As secret GitLab variables

  • The host and port endpoint of an external database,
  • The credentials of the external database,
  • The private key of the apps TLS certificate, and

Hard-coded in the repository

  • Application-specific configurations (colors, images, URL paths, etc).