Jafner/Jafner.net
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor Secrets Management #3

New Issue
Closed
opened 2024-08-12 16:35:04 -07:00 by Jafner · 12 comments
Jafner commented 2024-08-12 16:35:04 -07:00
Owner
Copy Link

Per #2

For each stack in homelab/fighter/config/*, we want to use compose secrets instead of opaquely injecting the contents of *_secrets.env into each container's environment.

Procedure

For each stack,

  1. Generate the secrets.env file:
for file in *_secrets.env; do 
    service="${file%%_*}_"
    for secret in $(cat $file); do
        key=${secret%%=*}
        value=${secret#*=}
        value=${value%\'}
        value=${value#\'}
        echo "$service$key=$value" >> secrets.env
    done 
done
  1. Generate the services > myapp > secrets configuration block:
echo "    secrets:"; for secret in $(cat secrets.env); do name=${secret%%=*}; echo "       - $name"; done
  1. Generate the services > myapp > environment block for secrets:
echo "    environment:"; for secret in $(cat secrets.env); do key=${secret#*_}; key=${key%%=*} ; echo "       $key: /run/secrets/${secret%%=*}"; done
  1. Generate services > myapp > environment block for non-secrets:
for file in $(find . -type f -name '*.env' -not -name '*_secrets.env' -exec basename {} \;); do echo "    environment:"; for envvar in $(cat $file); do key=${envvar%%=*}; value=${envvar#*=}; echo "      $key: $value"; done; done
  1. Generate the secrets: block:
echo "secrets:"; for secret in $(cat secrets.env); do name=${secret%%=*}; echo "  $name:\n    file: ./secrets/$name.txt"; done

  1. Put it all together to a refactored docker-compose.yml

  2. Populate our ./secrets/ directory:

mkdir -p ./secrets; for secret in $(cat secrets.env); do name=${secret%%=*}; value=${secret#*=}; echo $value > ./secrets/$name.txt; done
Per #2 For each stack in `homelab/fighter/config/*`, we want to use compose secrets instead of opaquely injecting the contents of `*_secrets.env` into each container's environment. ## Procedure For each stack, 1. Generate the `secrets.env` file: ```bash for file in *_secrets.env; do service="${file%%_*}_" for secret in $(cat $file); do key=${secret%%=*} value=${secret#*=} value=${value%\'} value=${value#\'} echo "$service$key=$value" >> secrets.env done done ``` 2. Generate the `services > myapp > secrets` configuration block: ```bash echo " secrets:"; for secret in $(cat secrets.env); do name=${secret%%=*}; echo " - $name"; done ``` 3. Generate the `services > myapp > environment` block for secrets: ```bash echo " environment:"; for secret in $(cat secrets.env); do key=${secret#*_}; key=${key%%=*} ; echo " $key: /run/secrets/${secret%%=*}"; done ``` 4. Generate `services > myapp > environment` block for non-secrets: ```bash for file in $(find . -type f -name '*.env' -not -name '*_secrets.env' -exec basename {} \;); do echo " environment:"; for envvar in $(cat $file); do key=${envvar%%=*}; value=${envvar#*=}; echo " $key: $value"; done; done ``` 5. Generate the `secrets:` block: ```bash echo "secrets:"; for secret in $(cat secrets.env); do name=${secret%%=*}; echo " $name:\n file: ./secrets/$name.txt"; done ``` 6. Put it all together to a refactored `docker-compose.yml` 7. Populate our `./secrets/` directory: ```bash mkdir -p ./secrets; for secret in $(cat secrets.env); do name=${secret%%=*}; value=${secret#*=}; echo $value > ./secrets/$name.txt; done ```
Jafner commented 2024-08-13 15:12:35 -07:00
Author
Owner
Copy Link

Ah, I recall why this solution kinda sucks.

Compose doesn't source the secret files. It only mounts files.

So when we say POSTGRES_USER: /run/secrets/postgres_POSTGRES_USER it sets the variable POSTGRES_USER to the literal string /run/secrets/postgres_POSTGRES_USER. Not very helpful.

Ah, I recall why this solution kinda sucks. Compose doesn't source the secret files. It only mounts files. So when we say `POSTGRES_USER: /run/secrets/postgres_POSTGRES_USER` it sets the variable `POSTGRES_USER` to the literal string `/run/secrets/postgres_POSTGRES_USER`. Not very helpful.
Jafner commented 2024-08-13 15:26:16 -07:00
Author
Owner
Copy Link

It seems that the standard approach to exporting secrets files to the container environment at startup is to inject a chained entrypoint script to read the /run/secrets/ directory and source each secret.

https://stackoverflow.com/questions/48094850/docker-stack-setting-environment-variable-from-secrets

We'll implement it for Keycloak to get things working and see how ugly it is.

It seems that the standard approach to exporting secrets files to the container environment at startup is to inject a chained entrypoint script to read the `/run/secrets/` directory and source each secret. https://stackoverflow.com/questions/48094850/docker-stack-setting-environment-variable-from-secrets We'll implement it for Keycloak to get things working and see how ugly it is.
Jafner commented 2024-08-14 00:20:55 -07:00
Author
Owner
Copy Link

Alright, I think we can use SOPS+age to store our secrets in the codebase, and use some scripting/automation to ease the burden of the encryption step.

This little command is pretty helpful for validating the compose file:
(export $(sops --decrypt --age ${SOPS_AGE_RECIPIENTS} secrets.enc.env | xargs); docker compose config)

And this is what we ran to bring up the stack:
export $(sops --decrypt --age ${SOPS_AGE_RECIPIENTS} secrets.enc.env | xargs); docker compose up -d

For these to work, we need the following environment variables set for SOPS to use:

  • SOPS_AGE_RECIPIENTS should be a comma-delimited list of pubkeys.
  • SOPS_AGE_KEY_FILE should be a path to the key file (e.g. ~/.age/key)
Alright, I think we can use SOPS+age to store our secrets in the codebase, and use some scripting/automation to ease the burden of the encryption step. This little command is pretty helpful for validating the compose file: `(export $(sops --decrypt --age ${SOPS_AGE_RECIPIENTS} secrets.enc.env | xargs); docker compose config)` And this is what we ran to bring up the stack: `export $(sops --decrypt --age ${SOPS_AGE_RECIPIENTS} secrets.enc.env | xargs); docker compose up -d` For these to work, we need the following environment variables set for SOPS to use: - `SOPS_AGE_RECIPIENTS` should be a comma-delimited list of pubkeys. - `SOPS_AGE_KEY_FILE` should be a path to the key file (e.g. `~/.age/key`)
Jafner commented 2024-08-14 00:21:30 -07:00
Author
Owner
Copy Link

Definitely have more work to do on the workflow.

Definitely have more work to do on the workflow.
Jafner commented 2024-08-14 11:54:56 -07:00
Author
Owner
Copy Link

Workflow still under development, but I think we're getting close to a simple, secure, system.

  • We have an .age-author-pubkeys file with a comma-separated list of pubkeys by whom all secrets should be decryptable. For now, that's one author (me).
  • For each host to which we want to deploy secrets, we have a .age-pubkey file at the root of that host's subdirectory (e.g. homelab/fighter/.age-pubkey).
  • When we encrypt a secret, we compose our SOPS_AGE_RECIPIENTS variable from the authors key list, and the pubkey of the host to which the secret belongs.
  • When we want to deploy a secret, only the host to which the secret belongs (and the authors) can decrypt the secret. This means our CI/CD runners will never be able to decrypt secrets themselves.

We still need to implement automation for this workflow with git filters, and script the setup process.

Workflow still under development, but I think we're getting close to a simple, secure, system. - We have an `.age-author-pubkeys` file with a comma-separated list of pubkeys by whom *all* secrets should be decryptable. For now, that's one author (me). - For each host to which we want to deploy secrets, we have a `.age-pubkey` file at the root of that host's subdirectory (e.g. `homelab/fighter/.age-pubkey`). - When we encrypt a secret, we compose our `SOPS_AGE_RECIPIENTS` variable from the authors key list, and the pubkey of the host to which the secret belongs. - When we want to deploy a secret, *only* the host to which the secret belongs (and the authors) can decrypt the secret. This means our CI/CD runners will never be able to decrypt secrets themselves. We still need to implement automation for this workflow with git filters, and script the setup process.
Jafner referenced this issue from a commit 2024-08-15 16:44:41 -07:00
#3 Delete test secret
Jafner referenced this issue from a commit 2024-08-15 16:45:31 -07:00
#3 Re-create test secret
Jafner referenced this issue from a commit 2024-08-16 12:19:54 -07:00
#3 Delete test secret
Jafner referenced this issue from a commit 2024-08-16 12:20:44 -07:00
#3 Re-introduce test secret
Jafner referenced this issue from a commit 2024-08-16 12:28:58 -07:00
#3 Delete testsecret
Jafner referenced this issue from a commit 2024-08-16 12:29:41 -07:00
#3 Re-create test secret
Jafner referenced this issue from a commit 2024-08-16 12:34:34 -07:00
#3 Delete test secret
Jafner referenced this issue from a commit 2024-08-16 12:43:37 -07:00
#3 Re-create test secret
Jafner referenced this issue from a commit 2024-08-16 12:44:52 -07:00
#3 Delete test secret
Jafner referenced this issue from a commit 2024-08-16 12:45:36 -07:00
#3 Re-create test secret
Jafner referenced this issue from a commit 2024-08-16 12:59:01 -07:00
#3 Delete test secret
Jafner referenced this issue from a commit 2024-08-16 13:00:33 -07:00
#3 Re-create test secret
Jafner referenced this issue from a commit 2024-08-16 13:22:32 -07:00
#3 Add second test secret
Jafner referenced this issue from a commit 2024-08-16 13:39:32 -07:00
#3 Delete test secrets
Jafner referenced this issue from a commit 2024-08-16 13:42:15 -07:00
#3 Re-create test secrets
Jafner referenced this issue from a commit 2024-08-16 14:33:32 -07:00
#3 chmod +x age-setup.sh
Jafner referenced this issue from a commit 2024-08-16 14:36:00 -07:00
#3 Delete test secrets
Jafner commented 2024-08-16 14:46:24 -07:00
Author
Owner
Copy Link

New procedure:

  1. Generate the consolidated environment blocks:
for file in $(find . -type f -name '*.env' -not -name '*_secrets.env' -exec basename {} \;); do echo "    environment:"; for envvar in $(cat $file); do key=${envvar%%=*}; value=${envvar#*=}; echo "      $key: $value"; done; done
  1. Generate the consolidated secrets file:
for file in *_secrets.env; do 
    service="${file%%_*}_"
    echo "    environment:"
    for secret in $(cat $file); do
        key=${secret%%=*}
        value=${secret#*=}
        value=${value%\'}
        value=${value#\'}
        echo "      $key: \${$service$key}"
        echo "$service$key=$value" >> secrets.env
    done 
done
  1. Generate environment blocks from secrets.env file:
for envvar in $(cat secrets.env); do key=${envvar
  1. Use export secrets.env; docker compose config to validate the stack.
  2. Use export secrets.env; docker compose up -d to bring up the stack.
New procedure: 1. Generate the consolidated environment blocks: ```bash for file in $(find . -type f -name '*.env' -not -name '*_secrets.env' -exec basename {} \;); do echo " environment:"; for envvar in $(cat $file); do key=${envvar%%=*}; value=${envvar#*=}; echo " $key: $value"; done; done ``` 2. Generate the consolidated secrets file: ```bash for file in *_secrets.env; do service="${file%%_*}_" echo " environment:" for secret in $(cat $file); do key=${secret%%=*} value=${secret#*=} value=${value%\'} value=${value#\'} echo " $key: \${$service$key}" echo "$service$key=$value" >> secrets.env done done ``` 3. Generate environment blocks from secrets.env file: ```bash for envvar in $(cat secrets.env); do key=${envvar ``` 3. Use `export secrets.env; docker compose config` to validate the stack. 4. Use `export secrets.env; docker compose up -d` to bring up the stack.
Jafner referenced this issue from a commit 2024-08-16 15:44:13 -07:00
#3 Show diffs in cleartext
Jafner commented 2024-08-16 15:55:50 -07:00
Author
Owner
Copy Link

We've run into a snag with our SOPS+age Git filter solution: the encrypted files are non-deterministic.

Everything works great when we write a secret in plaintext on our dev machine, git stage (and automatically encrypt), commit, and push it to the repo. It's stored in an encrypted state in our repo. Awesome.

And decrypting again on the server works great. We see only the decrypted file stored in plain text on the disk. No manual intervention required to decrypt the secrets.

However, our file is permanently "modified" on the server because it compares the two encrypted versions of the file (source encrypted by my private key, local encrypted by the server's private key) and will always see differences between them, as they do not result in the same encrypted file content.

We've run into a snag with our SOPS+age Git filter solution: the encrypted files are non-deterministic. Everything works great when we write a secret in plaintext on our dev machine, git stage (and automatically encrypt), commit, and push it to the repo. It's stored in an encrypted state in our repo. Awesome. And decrypting again on the server works great. We see only the decrypted file stored in plain text on the disk. No manual intervention required to decrypt the secrets. However, our file is permanently "modified" on the server because it compares the two encrypted versions of the file (source encrypted by my private key, local encrypted by the server's private key) and will always see differences between them, as they do not result in the same encrypted file content.
Jafner commented 2024-08-16 16:04:33 -07:00
Author
Owner
Copy Link

It may make sense to switch to another tool, such as git-crypt

It may make sense to switch to another tool, such as [git-crypt](https://github.com/AGWA/git-crypt)
Jafner commented 2024-08-16 16:50:19 -07:00
Author
Owner
Copy Link

git-crypt seems to only support GPG-based keys. A unique pain for my environment where the GPG agent seems to have gone senile, forcing me to delete a stale lockfile to perform any action.

Would much prefer a symmetric-key based approach.

Edit: oh, git-crypt does support symmetric-key encryption/decryption. We'll review that workflow.

git-crypt seems to *only* support GPG-based keys. A unique pain for my environment where the GPG agent seems to have gone senile, forcing me to delete a stale lockfile to perform any action. Would much prefer a symmetric-key based approach. Edit: oh, git-crypt *does* support symmetric-key encryption/decryption. We'll review that workflow.
Jafner commented 2024-08-16 17:08:50 -07:00
Author
Owner
Copy Link

Yeah this is a thousand times simpler. (Albeit with the limitations that entails).

  1. Give key to host. scp .git-crypt.key <user>@<host>:/path/to/repo/.git-crypt.key
  2. Host unlocks secrets. git-crypt unlock .git-crypt.key

That's the whole thing.

Yeah this is a thousand times simpler. (Albeit with the limitations that entails). 1. Give key to host. `scp .git-crypt.key <user>@<host>:/path/to/repo/.git-crypt.key` 2. Host unlocks secrets. `git-crypt unlock .git-crypt.key` That's the whole thing.
Jafner commented 2024-08-16 17:40:58 -07:00
Author
Owner
Copy Link

The process is so seamless than it can be difficult to verify that the committed files are, in fact, encrypted.

To verify that our staged files are encrypted, we can temporarily disable the git-crypt diff filter.

git -c diff.git-crypt.textconv=false diff --cached

Where:

  • git -c tells git to run with the following configure key-value overridding the default (as defined in the repo's .git/config
  • diff.git-crypt.textconv=false Disables our diff text conversion filter from git-crypt
  • and diff --cached compares our staged files to the previous commit.
The process is *so seamless* than it can be difficult to verify that the committed files are, in fact, encrypted. To verify that our staged files are encrypted, we can temporarily disable the git-crypt diff filter. `git -c diff.git-crypt.textconv=false diff --cached` Where: - `git -c` tells git to run with the following configure key-value overridding the default (as defined in the repo's `.git/config` - `diff.git-crypt.textconv=false` Disables our diff text conversion filter from git-crypt - and `diff --cached` compares our staged files to the previous commit.
Jafner referenced this issue from a commit 2024-08-16 17:41:29 -07:00
#3 Nuke sops
Jafner referenced this issue from a commit 2024-08-23 18:09:34 -07:00
#3 Set up sops at repo root
Jafner commented 2024-10-18 17:02:30 -07:00
Author
Owner
Copy Link

I'm pretty happy with where we're at. I suspect that the setup process has some friction, but we'll cross that bridge when we get to it.

As is, our workflow for secrets:

  1. When a file matching one of our .gitattributes patterns is checked in, it gets encrypted against the list of age recipients defined in .sops.yaml, requiring keys from any two of the four groups to decrypt.
  2. Our sops Git filter runs the .sops/encrypt-filter.sh script against each matched file before a secret is pushed to the remote.
  3. Our sops Git filter runs the .sops/decrypt-filter.sh script against each matched file after a secret is checked out locally.
  4. We can get the list of all files managed by sops with:
git ls-files |\
  git check-attr -a --stdin |\
  grep 'filter: sops' |\
  cut -d':' -f1
I'm pretty happy with where we're at. I suspect that the setup process has some friction, but we'll cross that bridge when we get to it. As is, our workflow for secrets: 1. When a file matching one of our `.gitattributes` patterns is checked in, it gets encrypted against the list of age recipients defined in `.sops.yaml`, requiring keys from any two of the four groups to decrypt. 2. Our `sops` Git filter runs the `.sops/encrypt-filter.sh` script against each matched file before a secret is pushed to the remote. 3. Our `sops` Git filter runs the `.sops/decrypt-filter.sh` script against each matched file after a secret is checked out locally. 4. We can get the list of all files managed by sops with: ```sh git ls-files |\ git check-attr -a --stdin |\ grep 'filter: sops' |\ cut -d':' -f1 ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Jafner/Jafner.net#3
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?