Docker Secrets With Letsencrypt

Although Docker 1.13 brings it’s usual load of features the most prohiminent one is certainly the secret management. There are many nice articles describing this feature so instead of presenting this feature, this article will focus on a solution to use Docker secrets with Letsencrypt.

Docker Secrets

Docker Secrets aim at providing a simple and secure way to store and use confidential data such as password, private keys, etc. The API is pretty simple:

• a new command docker secret to create, delete, list or inspect the secrets
• new options to docker service createand docker service update to control the container’s access to the secrets.

Secret Management

The secret creation is quite straightforward as docker secret create --help will show:

Usage:	docker secret create [OPTIONS] SECRET file|-

Create a secret from a file or STDIN as content

Options:
      --help         Print usage
  -l, --label list   Secret labels (default [])

When a secret is created Docker will guarantee to keep it encrypted when sending it to any remote node (encryption in transit) or when stored on the server filesystem (encryption at rest). The secrets are stored in the Raft log which is encrypted and replicated to all the managers in the cluster.

It is important to know that the log encryption is only performed by Docker 1.13 and above which means that if one of the cluster managers is running docker 1.12, the secrets can be stored uncrypted in this server filesystem.

The other secret subcommands are really as simple as this one with a couple of things worth noting:

1. Secrets are immutable: you can create a secret, delete a secret but you cannot change it.
2. A secret cannot be deleted while a service is using it.

Container Access to Secrets

Docker secrets implementation is strongly tied to swarm as the storage in the Raft log might suggest. As a consequence, secret are not available to plain old containers but only to services. At runtime, an in-memory filesystem will be mounted inside the container on /run/secrets and will contain one file for each secret the services has been given access to.

A service can be given access to a secret throught the --secret option of the docker service create command. For instance: docker service create --secret cartman will create a /run/secrets/cartman file inside the container. A more complex version of this option allows to specify the name of the secret file and specify the owner, group and the permissions of the secret file (have a look at Create a service with secret for more information).

Similarly, the docker service update commands have --service-rm and --service-add options (more information here).

Let’s Encrypt

Before talking about the integration with Docker secrets, let’s have a closer look at how Letsencrypt is actually working.

There are many ways of using Letsencrypt in this article I’ll be considering only the use of Cerbot. To add more restriction, I’ve only tested this with the standalone plugin while I’m pretty sure it’ll also work with the webroot plugin as well.

Obtaining a Certificate

If you are running a server with a public connection, getting the first certificate from Letsencrypt is quite easy:

docker run --rm -v $PWD/etc:/etc/letsencrypt -v $PWD/logs:/var/log/letsencrypt \
       -p 443:443 quay.io/letsencrypt/letsencrypt certonly \
       --standalone --staging \
       --non-interactive --email you@domain.com --agree-tos -d hostname

Here we added the --staging option in order to use the staging instead of the production platform.

In the current directory you’ll find a etc directory containing the certificates, the private keys and some miscellaneous configuration files used by letsencrypt. For instance the etc/live/hostname will contain everything you need to configure a web server with the newly generated certificates as indicated in the README:

This directory contains your keys and certificates.

`privkey.pem`  : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem`     : will break many server configurations, and should not be used
                 without reading further documentation (see link below).

We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.

Renewing the certificates

Letsencrypt create certificate with a very short validity: 90 days. The idea is to leverage the automated system to renew frequently your web servers’ certificate. Renewal is even easier that the creation as it can be done with:

docker run --rm -v $PWD/etc:/etc/letsencrypt -v $PWD/logs:/var/log/letsencrypt \
           -p 443:443 quay.io/letsencrypt/letsencrypt renew \
           --staging --force-renewal

Note the --force-renewal option in addition to --staging in order to renew the certificate even if we are not withing 30 days of the expiration date.

At this point you can notice that symbolic links in the etc/live/hostname directory have been updated to reflect the certificate renewal. If you look closer you’ll see that the files are merely symbolic links to files located under etc/archive

First Integration with Secrets

In order to test the integration I created a version of Nginx with SSL enabled and using key and certificate located under /run/secrets to be compatible with Docker Secrets:

You can build your own image or use the pre-built ggtools/test-nginx-ssl.

Creating the Secrets

We are going to create two secrets:

1. test_site.key from etc/live/<hostname>/privkey.pem
2. test_site.crt from etc/live/<hostname>/fullchain.pem

docker secret create test_site.key etc/live/hostname/privkey.pem
docker secret create test_site.crt etc/live/hostname/fullchain.pem

Creating the Service

The next step is to create a service using these secret:

docker service create -p 8443:443 --name nginx_test \
       --secret source=test_site.key,target=site.key \
       --secret source=test_site.crt,target=site.crt ggtools/test-nginx-ssl

Thanks to the source/target syntax, the secret names can be mapped to the file names expected by the image.

At this point, this should be working and the nginx container could be accessed from a browser. There should have been a security warning from the browser but as we used the staging environment this is completely normal. Should we celebrate then? Naaaah. There’s a small issue with this setting: upgrading the certificate will be complicated as secrets are immutable and cannot be deleted until removed from all services. That’ll be mean that when the certificate is renewed, both secrets will have to be removed from all services.

Improving the Integration

We have seen that the files from the etc/live/<hostname> directory are symbolic links to files in the etc/archive/<hostname> directory. If we look at this directory we’ll find that Letsencrypt is actually versioning the certificates and files. Which is exactly what we need to implement secret rotation.

Creating the Secrets

We are still going to create two secrets but we will use the files from etc/archive/<hostname> and will add a version to the secret names:

docker secret create test_site.key.1 etc/archive/hostname/privkey1.pem
docker secret create test_site.crt.1 etc/archive/hostname/fullchain1.pem

When renewing the certificate, Letsencrypt will create new files with a new version number. Will will then create new versionned secret:

docker secret create test_site.key.2 etc/archive/hostname/privkey2.pem
docker secret create test_site.crt.2 etc/archive/hostname/fullchain2.pem

Creating the Service

Not much difference at this point as we are only referencing the versionned secrets:

docker service create -p 8443:443 --name nginx_test \
       --secret source=test_site.key.1,target=site.key \
       --secret source=test_site.crt.1,target=site.crt ggtools/test-nginx-ssl

When renewing a certificate services could be updated one at the time since the old secrets will still exist:

docker service update --secret-rm test_site.key.1 --secret-rm test_site.crt.1 \
       --secret-add source=test_site.key.2,target=site.key \
       --secret-add source=test_site.crt.2,target=site.crt nginx_test

This command will be completed using the normal service update mechanism and will stop and restart the service’s tasks according to the update configuration.

Automation

Letsencrypt and Docker secrets have a very similar philosophy with versionned immutable files/secrets which means that setting up a (basic) automation would be really simple:

In a second step, we can inspect the service and automatically update the services using outdated certificates. The script will be a little be more complicated as we have to retrieve the full configuration to add the renewed certificates (target, uid, gid and mode). Also total automation might not be wanted as the service updates can be triggered at any time.

A second script will check if the services are using the latest version of the certificates. While the first script is pretty harmless since it only creates new secrets, this one should be used with more care as all updated services will be restarted. In order to mitigate this problem the script will only look for service with the le_auto label:

In a full automated environment, both scripts could be run from Cerbot using the --post-hook option.