Create docker machine with custom certificates

Have you ever tried to set up a new virtual machine with docker-machine using custom certificates? The docker-machine create command offers a couple of options to set custom certificates. However, to me, it was not clear, which option has what effect. This article discusses which certificates and keys are involved and which can be overridden with custom files.

Involved certificates and keys

Before I discuss the options of docker-machine, I’d like to show what certificates and keys are involved. Generally speaking, an HTTP connection secured with SSL/TLS can involve up to four different parties. Obviously, we need a server and a client. A client initiates the connection to the server. The server is in possession of a private key and a certificate. The certificate is issued by a third party (unless self-signed), the Certification Authority (CA). The CA asserts that the identity of the server is correct. The CA itself has a private key and a certificate (usually self- or cross-signed). The client trusts the CA to verify the identity of the server and therefore accepts the certificate sent by the server. A secure connection can be established. This is how most of the every-day HTTPS connections with your browser work.

When the docker-client (the docker command-line tool docker) connects to a remote docker daemon (dockerd) via a secure connection (port 2376 by default), the server (dockerd in this case) should also request a certificate from the client in order to reject unauthorized clients. The server only accepts clients if they have a certificate from a particular CA, here called Client-CA. You see, there are up to four parties involved when we use HTTPS client authentication. This is illustrated in the following figure.

The picture becomes even more complicated when you consider that the docker daemon becomes a client itself when it connects to a remote registry. In this case, we can use the above figure again, replacing docker-cli with dockerd and dockerd with the remote registry. This introduces three new parties: two CAs and the remote registry. It also introduces a new key and a new certificate for dockerd as a client. However, this complication is not the main focus of this article. I will treat dockerd mostly as the server in this article

The question is now, how can we create a new docker-machine with custom certificates.

Command-line options

The command-line tool docker-machine provides four general and one specific option to manage TLS credentials when creating a new machine. Version 0.13.0’s built-in help lists them as

# general options
--tls-ca-cert     CA to verify remotes against [$MACHINE_TLS_CA_CERT]
--tls-ca-key      Private key to generate certificates [$MACHINE_TLS_CA_KEY]
--tls-client-cert Client cert to use for TLS [$MACHINE_TLS_CLIENT_CERT]
--tls-client-key  Private key used in client TLS auth [$MACHINE_TLS_CLIENT_KEY]

# only for 'docker-machine create'
--tls-san option  Support extra SANs for TLS certs

By simply reading the built-in help, I didn’t understand which option overwrites which key or certificate in the above picture. After some investigations, I can now shed some light on this. The first key observation is that there is only a single CA issuing the server certificate and used to authenticate clients. The following figure summarizes the relation between the CA, the client and the server.

By default, docker-machine create uses the keys and certificates located at ~/.docker/machine/certs. This directory hosts the key and certificate for the CA (ca-key.pem and ca.pem), and the key and certificate for the client (key.pem and cert.pem). The client certificate was issued by this CA. The first four command line arguments can be used to shadow these files. If you add --tls-ca-cert for example, it will use the default files, except for the CA certificate. If this is a custom one, it most likely doesn’t match the default CA’s private key and therefore provisioning the machine will fail.

If you want to use different client credentials, use the --tls-client-cert and --tls-client-key options. Since this still uses the default CA, you need a certificate issued by the default CA, otherwise, you won’t be able to connect to the server.

If you want to use a different CA, use the --tls-ca-cert and --tls-ca-key arguments. The certificate should be a self-signed certificate matching the specified key. If you use a different CA, you also need to specify a custom client certificate. Otherwise, docker-machine uses the default ones which are not issued by the custom CA and won’t allow you to connect the server.

The private key and certificate of the server are always generated when you invoke docker-machine create. This is convenient because the machine’s IP address is not known in advance but must be included in the certificate. If your machine is accessible via other IP addresses or domain names use the --tls-san option to add additional Subject Alternative Names to the certificate. To issue the server certificate, docker-machine uses the CA’s private key: either the default key or the one specified via --tls-ca-key. If you specify a custom CA key, docker-machine will not store the key for you.

It is also interesting to note, that if you use the default options, docker-machine will use the default files for all machines. Machines created without any options share the same CA certificate, client key and client certificate.

In case you were wondering how to deal with custom certificates when the docker daemon itself becomes the HTTP client of a remote registry, it becomes a bit less user-friendly. You need to manage the certificates inside the virtual machine yourself. Check the docker help for more information.

Summary

The following table summarizes the relevant command-line arguments. The File column specifies where the machine-specific files corresponding to the command-line arguments are stored. All machine-specific files are located in ~/.docker/machine/machines/<MACHINE_NAME>. The Default column specifies the files that are used when the command-line arguments are omitted. The default files are stored in ~/.docker/machine/certs.

Option	File	Default	Description
--tls-ca-key	none	ca-key.pem	Private key of the Certification Authority. The server certificate is signed with this key. A custom key is not stored. Must be used in conjunction with --tls-ca-cert.
--tls-ca-cert	ca.pem	ca.pem	Self-signed certificates of the Certification Authority used for client and server authentication. Must be used in conjunction with --tls-ca-key.
--tls-client-key	cert.pem	cert.pem	Private key of the client. Must be used in conjunction with --tls-client-cert.
--tls-client-cert	key.pem	key.pem	Client certificate. Must be issued by the CA (custom or default) in order to connect to dockerd. Must be used in conjunction with --tls-client-key.
none	server-key.pem	none	Automatically generated.
(--tls-san)	server.pem	none	Automatically created certificate, issued by the CA (custom or default). The option --tls-san adds Subject Alternative Names to the certificate, in case your machine is reachable via different IP addresses or domain names.