Jafner/homelab
1
1
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor .env management #36

New Issue
Closed
opened 2022-07-15 12:39:10 -07:00 by Jafner · 41 comments
Jafner commented 2022-07-15 12:39:10 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Should be possible to replace a per-project templated .env file with an inheritance-based model.

References

Should be possible to replace a per-project templated `.env` file with an inheritance-based model. # References - [Docker - Environment Variables in Compose](https://docs.docker.com/compose/environment-variables/)
Jafner commented 2022-07-15 12:39:10 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

assigned to @Jafner

assigned to @Jafner
Jafner commented 2022-07-15 12:39:21 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

marked this issue as related to #34

marked this issue as related to #34
Jafner commented 2022-08-09 12:32:13 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Here's what I'm thinking:

  1. Create 3 tiers of .env file: homelab.env, host.env, and project.env.
  2. Populate homelab.env with variables that are relevant and correct for all hosts and applications. Examples: TZ=America/Los_Angeles, ADMIN_EMAIL=joey@jafner.net, PUID/PGID, and SMTP configuration variables.
  3. Populate host.env with variables that are relevant and correct for all applications on that host. Examples: DOCKER_DATA/DOCKER_CONFIG and media directories.
  4. Populate project.env with variables that are specific to an application or project.

Then, for each compose file, we define:

env_file:
  - ./project.env
  - ../../host.env
  - ../../../homelab.env
Here's what I'm thinking: 0. Create 3 tiers of `.env` file: `homelab.env`, `host.env`, and `project.env`. 1. Populate `homelab.env` with variables that are relevant and correct for all hosts and applications. Examples: `TZ=America/Los_Angeles`, `ADMIN_EMAIL=joey@jafner.net`, PUID/PGID, and SMTP configuration variables. 2. Populate `host.env` with variables that are relevant and correct for all applications on that host. Examples: `DOCKER_DATA`/`DOCKER_CONFIG` and media directories. 3. Populate `project.env` with variables that are specific to an application or project. Then, for each compose file, we define: ``` env_file: - ./project.env - ../../host.env - ../../../homelab.env ```
Jafner commented 2022-08-09 12:35:41 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

We'll roll this out for jafner.tools first.

We'll roll this out for jafner.tools first.
Jafner commented 2022-08-09 12:35:49 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Wish I had a dev env.

Wish I had a dev env.
Jafner commented 2022-08-09 12:38:24 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Actually, that's stupid. jafner.tools is supposed to be more reliable.
We'll build out the .env files, then recreate projects to use the new convention one at a time.

Actually, that's stupid. jafner.tools is supposed to be *more* reliable. We'll build out the `.env` files, then recreate projects to use the new convention one at a time.
Jafner commented 2022-08-09 13:20:15 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

I think the proper order is:

env_file:
  - ../../../homelab.env
  - ../../host.env
  - ./project.env

I believe this will create proper inheritance.

I think the proper order is: ``` env_file: - ../../../homelab.env - ../../host.env - ./project.env ``` I believe this will create proper inheritance.
Jafner commented 2022-08-09 13:23:54 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

We will be able to test this by inspecting resulting container env vars with: docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}'

We will be able to test this by inspecting resulting container env vars with: `docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}'`
Jafner commented 2022-08-09 15:15:14 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit f152d23a79

mentioned in commit f152d23a79fd5f26f2f931ea6bcbb5f94d3e56c5
Jafner commented 2022-08-09 15:15:14 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 6943756b1c

mentioned in commit 6943756b1c406e7e4b632a834203df0595adca0b
Jafner commented 2022-08-09 15:15:14 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 178335afc5

mentioned in commit 178335afc51b2eb1575370f8e80a5f9a8efb5873
Jafner commented 2022-08-09 15:15:14 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 33b63a05d4

mentioned in commit 33b63a05d4fe4d7d3bb46d3c2be555483ffc71aa
Jafner commented 2022-10-31 22:08:50 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

In preparation for #32, let's split out sensitive/secret env vars into their own secrets.env at the project level.

In preparation for #32, let's split out sensitive/secret env vars into their own secrets.env at the project level.
Jafner commented 2022-11-01 00:24:38 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 4b37285b8e

mentioned in commit 4b37285b8ed55e66322dd839c0e86fbf0c079929
Jafner commented 2022-11-01 00:29:45 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 74df7d409e

mentioned in commit 74df7d409e04b4708375590fe9ed17e8f9e06d24
Jafner commented 2022-11-01 00:30:27 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit c33c89e430

mentioned in commit c33c89e430fa5dffe1c57367b4451ad58c88d961
Jafner commented 2022-11-01 00:30:53 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit bf6ff38a63

mentioned in commit bf6ff38a6363b891054249fb7dbf0d8ba0576e31
Jafner commented 2022-11-01 00:31:47 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit da154beae5

mentioned in commit da154beae5d8385adfc1fd963a9802418e7cf40d
Jafner commented 2022-11-01 00:32:33 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 55aea52473

mentioned in commit 55aea5247335436d6f1b000a4b3c77fbccdb8887
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 0db2d341b0

mentioned in commit 0db2d341b0f38b6df7c24b4c6696700d3042b74b
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 4578384c1f

mentioned in commit 4578384c1f1e560f3b8689adb35e09f4b6f5d2b0
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit d3b3386c0d

mentioned in commit d3b3386c0d4492c283d41f03dfa60b0c9ae34d77
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 491e079c9a

mentioned in commit 491e079c9aa553631a403bad3ac6eff9874a5175
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 2ee6dfd884

mentioned in commit 2ee6dfd884a13d85e1c107db0e87af3233d1799e
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit a5bfb89aff

mentioned in commit a5bfb89aff05701551c7e8d530f325f899fcaa5b
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 345edae285

mentioned in commit 345edae285d79a3aa971689ad3339b0d126e5d11
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 0520696a1d

mentioned in commit 0520696a1d2b8c236df5ab0d484dfb29cfade596
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 69079e848f

mentioned in commit 69079e848f9bab7c14855d31d5f7561b49e27f9d
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 62ebf57f47

mentioned in commit 62ebf57f47225347343f4b93cc7bcf21aa3d8074
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit b99f2b6c60

mentioned in commit b99f2b6c60f9e4bfd9cc268e5ecd733d5398ef1f
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 2a780957bf

mentioned in commit 2a780957bf18e5582ce759a3b7b5e2e54fc6d3ce
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 0004eb3603

mentioned in commit 0004eb3603429ab408dc7dbbfc1645e362d9761c
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit d0dd8fc621

mentioned in commit d0dd8fc6210e4125a0888150e5f600d6c5585943
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 724a2a6932

mentioned in commit 724a2a6932efd0a531e4657ac007a262b90d99c4
Jafner commented 2022-11-01 00:36:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 2d670a59e0

mentioned in commit 2d670a59e02e5579647158bbec1db585de263286
Jafner commented 2022-11-01 00:51:03 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Updated all current services to use the following:

  1. No env vars at a greater than project level.
  2. Env vars shared across a project are declared in <project_name>.env.
  3. Env vars used by a single service are declared in <service_name>.env.
  4. Per #32, secrets are split into their own <project/service_name>_secrets.env file.

This system is intuitive, safe, and ensures all used env vars are easy to find, even without checking the running container. It is also easy to maintain.

Updated all current services to use the following: 1. No env vars at a greater than project level. 2. Env vars shared across a project are declared in `<project_name>.env`. 3. Env vars used by a single service are declared in `<service_name>.env`. 4. Per #32, secrets are split into their own `<project/service_name>_secrets.env` file. This system is intuitive, safe, and ensures all used env vars are easy to find, even without checking the running container. It is also easy to maintain.
Jafner commented 2022-11-01 11:45:11 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Re-opening with a logic error.
Env variables declared in files referenced by the "env_file:" compose key only pass information to the shell. While these variables are passed into the container's shell, they are not utilized if referenced in the compose file itself. See the result of docker-compose config for calibre-web below:

joey@joey-server:~/homelab/server/config/calibre-web$ docker-compose config
WARNING: The LIBRARY_DIR variable is not set. Defaulting to a blank string.
networks:
  web:
    external: true
    name: web
services:
  calibre-web-ace:
    container_name: calibre-web_ace
    environment:
      LIBRARY_DIR: /mnt/nas/calibre-web
      PGID: '1000'
      PUID: '1000'
      TZ: America/Los_Angeles
    image: linuxserver/calibre-web
    labels:
      traefik.http.routers.calibre-ace.rule: Host(`ace.calibre.jafner.net`)
      traefik.http.routers.calibre-ace.tls.certresolver: lets-encrypt
    logging:
      driver: loki
      options:
        keep-file: "true"
        loki-batch-size: '50'
        loki-retries: '1'
        loki-timeout: 2s
        loki-url: http://localhost:3100/loki/api/v1/push
    networks:
      web: null
    volumes:
    - calibre-web-ace_data:/config:rw
    - /ace:/books:rw
  calibre-web-rpg:
    container_name: calibre-web_rpg
    environment:
      LIBRARY_DIR: /mnt/nas/calibre-web
      PGID: '1000'
      PUID: '1000'
      TZ: America/Los_Angeles
    image: linuxserver/calibre-web
    labels:
      traefik.http.routers.calibre-rpg.rule: Host(`rpg.calibre.jafner.net`)
      traefik.http.routers.calibre-rpg.tls.certresolver: lets-encrypt
    logging:
      driver: loki
      options:
        keep-file: "true"
        loki-batch-size: '50'
        loki-retries: '1'
        loki-timeout: 2s
        loki-url: http://localhost:3100/loki/api/v1/push
    networks:
      web: null
    volumes:
    - calibre-web-rpg_data:/config:rw
    - /rpg:/books:rw
  calibre-web-sff:
    container_name: calibre-web_sff
    environment:
      LIBRARY_DIR: /mnt/nas/calibre-web
      PGID: '1000'
      PUID: '1000'
      TZ: America/Los_Angeles
    image: linuxserver/calibre-web
    labels:
      traefik.http.routers.calibre-sff.rule: Host(`sff.calibre.jafner.net`)
      traefik.http.routers.calibre-sff.tls.certresolver: lets-encrypt
    logging:
      driver: loki
      options:
        keep-file: "true"
        loki-batch-size: '50'
        loki-retries: '1'
        loki-timeout: 2s
        loki-url: http://localhost:3100/loki/api/v1/push
    networks:
      web: null
    volumes:
    - calibre-web-sff_data:/config:rw
    - /sff:/books:rw
version: '3'
volumes:
  calibre-web-ace_data: {}
  calibre-web-rpg_data: {}
  calibre-web-sff_data: {}
Re-opening with a logic error. Env variables declared in files referenced by the "env_file:" compose key only pass information to the shell. While these variables are passed into the container's shell, they are not utilized if referenced in the compose file itself. See the result of `docker-compose config` for [calibre-web](server/config/calibre-web/docker-compose.yml) below: ``` joey@joey-server:~/homelab/server/config/calibre-web$ docker-compose config WARNING: The LIBRARY_DIR variable is not set. Defaulting to a blank string. networks: web: external: true name: web services: calibre-web-ace: container_name: calibre-web_ace environment: LIBRARY_DIR: /mnt/nas/calibre-web PGID: '1000' PUID: '1000' TZ: America/Los_Angeles image: linuxserver/calibre-web labels: traefik.http.routers.calibre-ace.rule: Host(`ace.calibre.jafner.net`) traefik.http.routers.calibre-ace.tls.certresolver: lets-encrypt logging: driver: loki options: keep-file: "true" loki-batch-size: '50' loki-retries: '1' loki-timeout: 2s loki-url: http://localhost:3100/loki/api/v1/push networks: web: null volumes: - calibre-web-ace_data:/config:rw - /ace:/books:rw calibre-web-rpg: container_name: calibre-web_rpg environment: LIBRARY_DIR: /mnt/nas/calibre-web PGID: '1000' PUID: '1000' TZ: America/Los_Angeles image: linuxserver/calibre-web labels: traefik.http.routers.calibre-rpg.rule: Host(`rpg.calibre.jafner.net`) traefik.http.routers.calibre-rpg.tls.certresolver: lets-encrypt logging: driver: loki options: keep-file: "true" loki-batch-size: '50' loki-retries: '1' loki-timeout: 2s loki-url: http://localhost:3100/loki/api/v1/push networks: web: null volumes: - calibre-web-rpg_data:/config:rw - /rpg:/books:rw calibre-web-sff: container_name: calibre-web_sff environment: LIBRARY_DIR: /mnt/nas/calibre-web PGID: '1000' PUID: '1000' TZ: America/Los_Angeles image: linuxserver/calibre-web labels: traefik.http.routers.calibre-sff.rule: Host(`sff.calibre.jafner.net`) traefik.http.routers.calibre-sff.tls.certresolver: lets-encrypt logging: driver: loki options: keep-file: "true" loki-batch-size: '50' loki-retries: '1' loki-timeout: 2s loki-url: http://localhost:3100/loki/api/v1/push networks: web: null volumes: - calibre-web-sff_data:/config:rw - /sff:/books:rw version: '3' volumes: calibre-web-ace_data: {} calibre-web-rpg_data: {} calibre-web-sff_data: {} ```
Jafner commented 2022-11-01 11:55:50 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

Ultimately, this means we'll need to handle volume variables differently from traditional env vars.

I can think of a few ways to do this:

  1. Fully de-variablize paths. Hard-code all volume maps.
  2. Store volume vars in the .env file.

I'm inclined toward the latter.

Ultimately, this means we'll need to handle volume variables differently from traditional env vars. I can think of a few ways to do this: 1. Fully de-variablize paths. Hard-code all volume maps. 2. Store volume vars in the `.env` file. I'm inclined toward the latter.
Jafner commented 2022-11-01 12:13:02 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

mentioned in commit 306bd8d132

mentioned in commit 306bd8d1322d8dcae13265b3d42ab5748583b335
Jafner commented 2022-11-01 13:50:01 -07:00 (Migrated from gitlab.jafner.net)
Copy Link

All volume vars are now stored in the project's .env file, which is not passed into the container. It is only used in the compose files themselves.

All volume vars are now stored in the project's `.env` file, which is not passed into the container. It is only used in the compose files themselves.
Jafner commented 2022-11-23 09:37:12 -08:00 (Migrated from gitlab.jafner.net)
Copy Link

This is done. This solution has been tested and working for 3 weeks.

This is done. This solution has been tested and working for 3 weeks.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Jafner/homelab#36
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?