Design CD for Compose Stacks #7

New Issue

Jafner · 2024-08-19T09:57:46-07:00

Jafner commented

2024-08-19 09:57:46 -07:00

Presently, we deploy updates manually. The process is relatively consistent, and can sometimes accelerate debugging. But with automation we can accelerate most of the development process, and build a foundation for automated post-deployment support.

Below are some considerations for the design of the pipeline:

CI should block deployment of any unencrypted secrets. If a file listed in .gitattributes with an encryption filter renders as unencrypted in the runner, it should immediately cancel the job and send an urgent notification to remediate the leaked secret(s).
I would like Stacks to be deployable without Git on the host. Treating each Stack as a "package" follows best-practice of loosely-coupling our services. More on this in a future comment.
Look into observability tooling with Docker automatic service discovery (like Traefik and Homepage have). Would like to have automated availability monitoring.
With automated deployment, we can eliminate the need for some of the clunky workflow streamlining strategies we've implemented for working on Fighter, such as repo-level secret encryption/decryption with git-crypt. Instead, we can focus on workflow improvements on development machines and not work about implementing the same tools on the host.

Presently, we deploy updates manually. The process is relatively consistent, and can sometimes accelerate debugging. But with automation we can accelerate most of the development process, and build a foundation for automated post-deployment support. Below are some considerations for the design of the pipeline: - CI should block deployment of any unencrypted secrets. If a file listed in `.gitattributes` with an encryption filter renders as unencrypted in the runner, it should immediately cancel the job and send an urgent notification to remediate the leaked secret(s). - I would like Stacks to be deployable without Git on the host. Treating each Stack as a "package" follows best-practice of loosely-coupling our services. More on this in a future comment. - Look into observability tooling with Docker automatic service discovery (like Traefik and Homepage have). Would like to have automated availability monitoring. - With automated deployment, we can eliminate the need for some of the clunky workflow streamlining strategies we've implemented for working on Fighter, such as repo-level secret encryption/decryption with git-crypt. Instead, we can focus on workflow improvements on development machines and not work about implementing the same tools on the host.

Jafner commented

2024-08-19 10:06:27 -07:00

Stack Package Components

Below are some of the design considerations for a "package" schema for Stacks:

The core of the Stack is a docker-compose.yml file.
Volume mounts are defined absolutely in .env because that file name is automatically sourced when bringing up the Compose stack. (E.g. DOCKER_DATA=/mnt/nas/docker_data/my_stack).
Secrets are defined in a secrets.env file, which is encrypted in the repo, and decrypted for the deployment process. Alternatively, secrets may be defined in the Compose file itself and encrypted with Sops. Discussion in #3.
Stacks do not have dependencies. Best practice is for the handling of dependency logic to be defined in the Stack's applications. We may need to find a workaround for dependency on specific host resources, such as NAS availability.
Stacks may have a deploy/ subdirectory with resources for deploying the Stack, such as variables defined for the CI or CD scripts, or helper scripts. These would be defined in deploy.env. Examples include: DEPLOY_LOGS_PATH, DEPLOY_SMB_SHARES, DEPLOY_S3_ENDPOINTS, DEPLOY_HTTP_ENDPOINTS, and more.

### Stack Package Components Below are some of the design considerations for a "package" schema for Stacks: - The core of the Stack is a `docker-compose.yml` file. - Volume mounts are defined absolutely in `.env` because that file name is automatically sourced when bringing up the Compose stack. (E.g. `DOCKER_DATA=/mnt/nas/docker_data/my_stack`). - Secrets are defined in a `secrets.env` file, which is encrypted in the repo, and decrypted for the deployment process. Alternatively, secrets may be defined in the Compose file itself and encrypted with Sops. Discussion in #3. - Stacks do not have dependencies. Best practice is for the handling of dependency logic to be defined in the Stack's applications. We may need to find a workaround for dependency on specific host resources, such as NAS availability. - Stacks may have a `deploy/` subdirectory with resources for deploying the Stack, such as variables defined for the CI or CD scripts, or helper scripts. These would be defined in `deploy.env`. Examples include: `DEPLOY_LOGS_PATH`, `DEPLOY_SMB_SHARES`, `DEPLOY_S3_ENDPOINTS`, `DEPLOY_HTTP_ENDPOINTS`, and more.

Jafner commented

2024-08-19 10:38:42 -07:00

Deployment Pipeline

Zip/tar the package folder.
scp the package file onto the host.
ssh into the host, extract the package, decrypt the secrets file using the host's key, pull the latest version of the used images. Run docker compose config and pipe the output into a checksum. Compare the checksum against the one defined in the package. If they differ, abort and report. Otherwise,
docker compose up -d --force-recreate > $CD_LOG_FILE 2&>1

### Deployment Pipeline 1. Zip/tar the package folder. 2. `scp` the package file onto the host. 3. `ssh` into the host, extract the package, decrypt the secrets file using the host's key, pull the latest version of the used images. Run `docker compose config` and pipe the output into a checksum. Compare the checksum against the one defined in the package. If they differ, abort and report. Otherwise, 4. `docker compose up -d --force-recreate > $CD_LOG_FILE 2&>1`

Jafner referenced this issue

2024-08-19 12:43:41 -07:00

Improve Management Plane: Secrets Management #2

Jafner added this to the Automated Deployment milestone 2024-10-20 15:06:17 -07:00

Jafner commented

2024-10-20 15:07:36 -07:00

Moved this to the automation milestone, but we're unlikely to continue development of Stacks as we move toward K8s for our core infrastructure.

Sign in to join this conversation.