Improve Management Plane: Secrets Management #2
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Currently our secrets are stored in plain text on the server on which they are needed, and git ignored.
We have some systems for keeping secrets out of version control, but secret management is essentially manual.
Establish Specifications
Evaluate Solutions
Intermediate step before adding tools to the toolchain: flatten our environment files into the stack compose file and use docker secrets to set values explicitly.
Example for Keycloak:
Before:
homelab/fighter/config/keycloak/treecat docker-compose.ymlAfter:
/homelab/fighter/config/keycloak/treecat docker-compose.ymlThe latter consolidates all configuration details (except the actual secret values) into the compose.
It would be a bit inconvenient to manually create each of these text files one at a time.
So instead we can keep a stack-level
secrets.envfile (or similar name) populated with the file names and values of each secret, then run a one-liner to create each secret file during deployment.One-liner:
mkdir -p ./secrets; for secret in $(cat secrets.env); do name=${secret%%=*}; value=${secret#*=}; echo $value > ./secrets/$name.txt; doneThis strategy integrates nicely with CI/CD if we want to use (for example) SOPS + age to store the
secrets.envfile with its values (but not keys) encrypted in the repository. At the same time, we can manage the secrets manually without much hassle so long as we keep thesecrets.envfile out of source control.We can accelerate the migration with a script to parse our existing
_secrets.envfiles and consolidate them into a newsecrets.envstack-level file.Although this doesn't help us refactor our compose files...
After getting some experience with sops and age, as well as git-crypt, I think that adding a secrets management service (such as Vault or Delinea) would offer little value while we've got only a single developer on the team.
If Docker/Compose secrets had tighter integration and our environment had more moving parts, it may make sense to centralize our secrets and take advantage of easier rotation and sharing, but presently all (?) of our secrets are simple configuration parameters passed into Stacks at launch, and are extremely simple to manage.
With that in mind, and with some further reflection on the discussions in #3 and #7, here's the management plane I think we've settled on for this chapter:
secrets.envas simple key-value pairs that we want to use in our Compose stack.decrypted~$file_namefile at the location of the secret.Managing Decryption Keys
Our secrets are accessible through the Docker host onto which they are deployed (via
docker inspecting env vars). So efforts to obfuscate the values on the host are mostly pointless. However, it would be preferred to avoid keeping a decryption key on the host.The CI/CD environment has a key, and each deploy target (host) has a key. I believe sops supports configuring a secret to require more than one key to fully decrypt the secret, which would mean that we could disallow the host or runner from decrypting the secret alone.
For this, we would need to ensure the deploy host has a private key located at a well-known path, and a pubkey listed in the repository. We would also need to configure our Actions environment with an age keypair.
Edit: see sops key groups for reference on configuring multi-key decryption.
Secrets Modification Workflow Considerations
Consider the following:
One of the guiding principles for this repo is declarative-ness. As such, using sops to store our secrets securely and declaratively is probably the simplest valid management plane for us.
We're closing this issue until further change is necessary.