In part 1, I set up an automated deployment process for a simple app on Kubernetes. There was a single environment in which to deploy the app. In the real world, you’d at least need an environment to test changes (
staging) in addition to the actual
production environment. Likely, you’ll also need a
development environment for developers to be able to run experimental changes.
In this post, I explore deploying a sample app in multiple environments. In the process, I also learned about how Kubernetes supports managing configuration data (
ConfigMaps) and secrets.
The app in part 1 was too simple. To make it a little more representative of the real world, I modified the app to serve two API endpoints -
POST /employees and
GET /employees. It stores
employees in a database. The database driver and connection string are taken from environment variables. This will allow us to test with sqlite locally (e.g see unit test) and use postgres in the cluster. Find the sample app on github.
Kubernetes supports namespaces which provide a named scope for kubernetes resources. Most resources can be created within a namespace 1. There is a
default namespace which is used by default (duh) when you don’t specify a namespace. This looks like it fits our use case of multiple isolated environments. Let’s go ahead and create two namespaces
WARNING: Even though namespaces may functionally isolate resources, the underlying compute, I/O and storage are shared – they’re part of the same cluster after all. As such, I wouldn’t recommend running the production environment in a namespace. Make a separate cluster for it. Take this post as a learning exercise, not what you’d want to actually do in production.
Setting up the DB
Let’s spin up a DigitalOcean managed Postgres database. The cheapest option at $15/month is sufficient for our toy use case. Make sure we only allow incoming connections from our k8s cluster 2. This is pretty intuitive to do from the UI. DO creates a default user/database and enables SSL by default which is nice. After it’s done spinning up the DB, you should get a set of parameters to connect to it including the password and the root CA certificate used to sign the DB certificate – we’ll need these, so download them.
To isolate environments, we’ll create separate logical DBs within postgres 3. Again, in a real production use case, you’ll want to spin up a separate DB instance for each environment. First, let’s create a user and logical DB for the
staging environment. Remember, we configured our instance to only allow connections from the k8s cluster, so to do this we’ll need to run a pod in the cluster:
Notice that the command is similar to
docker run which is nice to run a one-off pod in the cluster for admin tasks like this.
This drops you into a postgres shell. Create a postgres role for the API user, the database and
employees table, then grant the API user access:
Exit the shell, and you’ll see a message that the pod was deleted (the
--rm flag ensures this).
Great, the DB is now ready for use in the
Securing the database connection
You’ll notice when we connected to the DB in the above commands, we didn’t specify anything related to SSL or verifying certificates. The default sslmode is
prefer which means it’ll try a SSL connection and if that fails, it’ll try a non-SSL connection. This is dangerously insecure because it leaves us vulnerable to downgrade attacks. The
require mode is not sufficient either, since it does not verify the CA certificate if left unspecified; this opens us up to man-in-the-middle attacks. The only really secure mode is
verify-full which verifies the CA certificate as well as the hostname. Luckily, DO has provided us with the CA cert, so we need to pass
"sslmode=verify-full sslrootcert=<path to CA cert>".
ConfigMap for storing certificate
So, now our problem is, how do we get the certificate accessible to the application pod? Enter configmaps. A configmap is a bag of configuration data that can be consumed by pods in various ways. They can be scoped to a namespace, so we can specify different values for
production. A thing to keep in mind is that configmaps are not for secret data – so don’t store your API tokens and secret keys in configmaps.
NOTE: The root CA certificate is not secret data. It does not contain the private key.
Let’s create a configmap scoped to
staging from the CA certificate:
Now, see that the configmap was created with a single key
cacertificate.crt with the value being the certificate data:
Mounting the ConfigMap
Now, let’s specify a deployment for our app, specifying the
DB_CONNECTION_STRING environment variables:
The interesting bits here are
volumeMounts. You can specify a volume with a configmap as its source – each key in the configmap becomes a file. In
volumeMounts, you specify where in the filesystem you want to mount it (
/dbparams). With the mount in place, we can pass
sslrootcert=/dbparams/cacertificate.crt in the connection string.
kubectl apply the deployment and our app now has a secure connection to the DB.
Securing the password
Notice that we’re passing around the DB password in plain-text – this is bad practice. The yaml resources will be checked into source control as part of our GitOps workflow, and checking in secrets is a bad, bad idea.
Kubernetes can store secrets for us and pass them safely to pods. A vanilla K8s install stores secrets unencrypted (which is batshit crazy IMO). Most managed providers including DO K8s encrypts secrets by default.
Let’s store our DB credentials in a secret:
Similar to configmaps, there are various ways to consume secrets in pods. We’ll use environment variables. Let’s update the
We’ve defined two env variables
DB_PASSWORD which source their values from the secret
db-secret. Simple enough.
Dependent environment variables
Notice that our app doesn’t actually take
DB_PASSWORD environment variables; it takes a DB connection string. We’ll need to somehow refer to the user and password variables in
DB_CONNECTION_STRING. Fortunately, Kubernetes has just the thing - dependent environment variables. If you use $(VAR), K8s will substitute the value of $VAR. Let’s define the connection string then:
We’re now securely passing the password to our application.
Environment variables from ConfigMap
There are a couple more kinks to iron out. The
dbname is hard-coded, so it wouldn’t work in the
production environment. We’ll need to extract it into an environment variable. In the real world, the db host/port would also be different for staging vs production, so that needs to be parameterized as well. Earlier, we saw how to mount a
ConfigMap as a volume; now let’s see how to consume it as environment variables. First, let’s edit our
This opens the configmap in your $EDITOR. Go ahead and add a key
db_name with value “acmedb_staging” and save it. We can now create an environment variable sourced from this key in
Let’s do the same for DB_HOST and DB_PORT. Now we have a nicely parameterized deployment resource that can be applied to any environment. The full deployment.yaml looks like:
Deploying to production
We’ve already done all the heavy lifting to make our app ready to deploy in any environment. We’ll need to create the
acmedb_prod logical DB and with the
apiuser_prod postgres user similar to
staging. We’ll also need to create the
dbparams configmap and the username/password secret in the
production namespace (remember configmaps are scoped to namespaces):
Now, we can apply the same deployment resource without changes!
We learned about namespaces to create scoped resources. We learned how to utilize configmaps to cleanly separate code from configuration, allowing us to easily deploy in multiple environments. Additionally, we utilized secrets to securely pass confidential data to our app.
That said, I wouldn’t use namespaces as an isolation mechanism in real production as the underlying compute is shared; I’d create a separate cluster instead. Maybe dev/staging can use a shared cluster with namespaces.
Re: secrets, I’d be extremely careful to make sure that they’re configured to be encrypted with KMS (which is the default in Google Kubernetes Engine). To be honest, I wouldn’t recommend using secrets in the DO managed K8s, as there is little clarity on which encryption provider is used.
Additionally, Kubernetes secrets are static. Rotating them requires re-deploying your app to pick up the new values. There are 3rd party projects to automate this process, but that feels quite hacky to me. For short-lived/dynamic secret support, you’ll want to skip K8s secrets altogether and utilize something like Vault.
In future blog posts, I’ll explore extending my automated deployment process in part 1 to a proper CI/CD pipeline across staging and production environments.