Jafner/Jafner.net
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity

Separate service data from host #15

New Issue
Open
opened 2024-10-19 20:50:37 -07:00 by Jafner · 5 comments
Jafner commented 2024-10-19 20:50:37 -07:00
Owner
Copy Link

To facilitate our migration to a distributed system architecture, we need to decouple service data from the host the service is presently running on. This applies to Fighter and Druid.

Fighter

There are four places we keep service data on Fighter:

  • /home/admin/data/$STACK/$SERVICE
  • Beside the service's config files.
  • /mnt/iscsi/{barbarian|paladin}
  • /mnt/nas/{barbarian|paladin}

Druid

I think we moved all our service data for Druid to the data volume, but I don't recall its path.

Action Items

  • Catalog data to be moved.
  • Discover solutions for data storage suitable for distributed workloads.
To facilitate our migration to a distributed system architecture, we need to decouple service data from the host the service is presently running on. This applies to Fighter and Druid. ### Fighter There are four places we keep service data on Fighter: - `/home/admin/data/$STACK/$SERVICE` - Beside the service's config files. - `/mnt/iscsi/{barbarian|paladin}` - `/mnt/nas/{barbarian|paladin}` ### Druid I think we moved all our service data for Druid to the data volume, but I don't recall its path. ## Action Items - [x] Catalog data to be moved. - [ ] Discover solutions for data storage suitable for distributed workloads.
Jafner commented 2024-10-22 15:00:45 -07:00
Author
Owner
Copy Link

Catalog data to be moved.

We got a list of all host-side paths for data mounts in containers with docker inspect $(docker ps -aq) | jq .[].Mounts | grep Source | sort -u | cut -d'"' -f4

/etc/localtime
/home/admin/data/autopirate/bazarr
/home/admin/data/autopirate/sabnzbd_config
/home/admin/data/books/audiobookshelf
/home/admin/data/books/audiobookshelf-metadata
/home/admin/data/books/calibre
/home/admin/data/books/calibre/metadata.db
/home/admin/data/books/calibre-web
/home/admin/data/books/sabnzbd
/home/admin/data/homepage/tasmota/logs
/home/admin/data/jellyfin/jellyfin
/home/admin/data/monitoring/custom.ini
/home/admin/data/monitoring/grafana
/home/admin/data/monitoring/prometheus
/home/admin/data/monitoring/uptime-kuma
/home/admin/data/navidrome
/home/admin/data/plex/ombi
/home/admin/data/plex/plex
/home/admin/data/qbittorrent
/home/admin/data/stash
/home/admin/data/traefik/acme-dns01.json
/home/admin/data/traefik/acme.json
/home/admin/data/unifi_controller/config
/home/admin/data/warpgate
/home/admin/data/wireguard/wg-easy
/home/admin/Jafner.net/homelab/fighter/config/ddns/ddclient
/home/admin/Jafner.net/homelab/fighter/config/homepage/config
/home/admin/Jafner.net/homelab/fighter/config/homepage/icons
/home/admin/Jafner.net/homelab/fighter/config/monitoring/exporter-ping/config.yml
/home/admin/Jafner.net/homelab/fighter/config/monitoring/prometheus.yml
/home/admin/Jafner.net/homelab/fighter/config/traefik/config
/home/admin/Jafner.net/homelab/fighter/config/traefik/traefik.yaml
/home/admin/stacks/gitea-runner/config.yaml
/home/admin/stacks/gitea-runner/registration.token
/home/admin/stacks/home-assistant/mosquitto.conf
/home/admin/stacks/home-assistant/mosquitto.passwd
/mnt/iscsi/barbarian/autopirate/jackett_config
/mnt/iscsi/barbarian/autopirate/lidarr_config
/mnt/iscsi/barbarian/autopirate/nzbhydra2_config
/mnt/iscsi/barbarian/autopirate/radarr_config
/mnt/iscsi/barbarian/autopirate/readarr_config
/mnt/iscsi/barbarian/autopirate/sonarr_config
/mnt/iscsi/barbarian/autopirate/tdarr/configs
/mnt/iscsi/barbarian/autopirate/tdarr/logs
/mnt/iscsi/barbarian/autopirate/tdarr/server
/mnt/iscsi/barbarian/home-assistant/home-assistant
/mnt/iscsi/barbarian/home-assistant/mosquitto
/mnt/iscsi/barbarian/nextcloud/config
/mnt/iscsi/barbarian/nextcloud/data
/mnt/iscsi/barbarian/nextcloud/mariadb
/mnt/iscsi/barbarian/nextcloud/redis
/mnt/iscsi/barbarian/send/uploads
/mnt/iscsi/barbarian/tdarr/transcode_cache
/mnt/iscsi/barbarian/torrenting/NZB
/mnt/iscsi/barbarian/torrenting/NZB_incomplete
/mnt/iscsi/barbarian/zipline/postgres
/mnt/iscsi/barbarian/zipline/zipline/public
/mnt/iscsi/barbarian/zipline/zipline/uploads
/mnt/iscsi/paladin/stash/cache
/mnt/iscsi/paladin/stash/generated
/mnt/iscsi/paladin/stash/metadata
/mnt/nas/3DPrinting/Model Library/VanDAM
/mnt/nas/av/Galleries
/mnt/nas/av/Videos
/mnt/nas/books
/mnt/nas/books/audiobooks
/mnt/nas/books/ebooks
/mnt/nas/calibre-web/rpg
/mnt/nas/calibre-web/sff
/mnt/nas/movies
/mnt/nas/music
/mnt/nas/shows
/mnt/nas/torrenting
/run/dbus
/var/lib/docker/volumes/026b6b91cdf17575f0948aff512fa381cd39396bf4ecbff4d82cb734567b4e5b/_data
/var/lib/docker/volumes/6216013b5413d62e175974f8f0d923309ea51681f0f7a8205e46fae22f8de8c5/_data
/var/lib/docker/volumes/calibre-web_calibre-web-rpg_data/_data
/var/lib/docker/volumes/calibre-web_calibre-web-sff_data/_data
/var/lib/docker/volumes/cdd5565328632ce00ed158eb1c019f567bcc701588ce583e42efbfc55095ec13/_data
/var/lib/docker/volumes/keycloak_postgres_data/_data
/var/lib/docker/volumes/manyfold_postgres_data/_data
/var/run/docker.sock

A few of these are not real data paths (/run/dbus or /var/run/docker.sock), and some are remote filesystems (/mnt/*). The rest are local data stores that should be moved. Next comment will be the filtered list of paths to move.

## Catalog data to be moved. We got a list of all host-side paths for data mounts in containers with `docker inspect $(docker ps -aq) | jq .[].Mounts | grep Source | sort -u | cut -d'"' -f4` ``` /etc/localtime /home/admin/data/autopirate/bazarr /home/admin/data/autopirate/sabnzbd_config /home/admin/data/books/audiobookshelf /home/admin/data/books/audiobookshelf-metadata /home/admin/data/books/calibre /home/admin/data/books/calibre/metadata.db /home/admin/data/books/calibre-web /home/admin/data/books/sabnzbd /home/admin/data/homepage/tasmota/logs /home/admin/data/jellyfin/jellyfin /home/admin/data/monitoring/custom.ini /home/admin/data/monitoring/grafana /home/admin/data/monitoring/prometheus /home/admin/data/monitoring/uptime-kuma /home/admin/data/navidrome /home/admin/data/plex/ombi /home/admin/data/plex/plex /home/admin/data/qbittorrent /home/admin/data/stash /home/admin/data/traefik/acme-dns01.json /home/admin/data/traefik/acme.json /home/admin/data/unifi_controller/config /home/admin/data/warpgate /home/admin/data/wireguard/wg-easy /home/admin/Jafner.net/homelab/fighter/config/ddns/ddclient /home/admin/Jafner.net/homelab/fighter/config/homepage/config /home/admin/Jafner.net/homelab/fighter/config/homepage/icons /home/admin/Jafner.net/homelab/fighter/config/monitoring/exporter-ping/config.yml /home/admin/Jafner.net/homelab/fighter/config/monitoring/prometheus.yml /home/admin/Jafner.net/homelab/fighter/config/traefik/config /home/admin/Jafner.net/homelab/fighter/config/traefik/traefik.yaml /home/admin/stacks/gitea-runner/config.yaml /home/admin/stacks/gitea-runner/registration.token /home/admin/stacks/home-assistant/mosquitto.conf /home/admin/stacks/home-assistant/mosquitto.passwd /mnt/iscsi/barbarian/autopirate/jackett_config /mnt/iscsi/barbarian/autopirate/lidarr_config /mnt/iscsi/barbarian/autopirate/nzbhydra2_config /mnt/iscsi/barbarian/autopirate/radarr_config /mnt/iscsi/barbarian/autopirate/readarr_config /mnt/iscsi/barbarian/autopirate/sonarr_config /mnt/iscsi/barbarian/autopirate/tdarr/configs /mnt/iscsi/barbarian/autopirate/tdarr/logs /mnt/iscsi/barbarian/autopirate/tdarr/server /mnt/iscsi/barbarian/home-assistant/home-assistant /mnt/iscsi/barbarian/home-assistant/mosquitto /mnt/iscsi/barbarian/nextcloud/config /mnt/iscsi/barbarian/nextcloud/data /mnt/iscsi/barbarian/nextcloud/mariadb /mnt/iscsi/barbarian/nextcloud/redis /mnt/iscsi/barbarian/send/uploads /mnt/iscsi/barbarian/tdarr/transcode_cache /mnt/iscsi/barbarian/torrenting/NZB /mnt/iscsi/barbarian/torrenting/NZB_incomplete /mnt/iscsi/barbarian/zipline/postgres /mnt/iscsi/barbarian/zipline/zipline/public /mnt/iscsi/barbarian/zipline/zipline/uploads /mnt/iscsi/paladin/stash/cache /mnt/iscsi/paladin/stash/generated /mnt/iscsi/paladin/stash/metadata /mnt/nas/3DPrinting/Model Library/VanDAM /mnt/nas/av/Galleries /mnt/nas/av/Videos /mnt/nas/books /mnt/nas/books/audiobooks /mnt/nas/books/ebooks /mnt/nas/calibre-web/rpg /mnt/nas/calibre-web/sff /mnt/nas/movies /mnt/nas/music /mnt/nas/shows /mnt/nas/torrenting /run/dbus /var/lib/docker/volumes/026b6b91cdf17575f0948aff512fa381cd39396bf4ecbff4d82cb734567b4e5b/_data /var/lib/docker/volumes/6216013b5413d62e175974f8f0d923309ea51681f0f7a8205e46fae22f8de8c5/_data /var/lib/docker/volumes/calibre-web_calibre-web-rpg_data/_data /var/lib/docker/volumes/calibre-web_calibre-web-sff_data/_data /var/lib/docker/volumes/cdd5565328632ce00ed158eb1c019f567bcc701588ce583e42efbfc55095ec13/_data /var/lib/docker/volumes/keycloak_postgres_data/_data /var/lib/docker/volumes/manyfold_postgres_data/_data /var/run/docker.sock ``` A few of these are not real data paths (`/run/dbus` or `/var/run/docker.sock`), and some are remote filesystems (`/mnt/*`). The rest are local data stores that should be moved. Next comment will be the filtered list of paths to move.
Jafner commented 2024-10-22 15:01:29 -07:00
Author
Owner
Copy Link
/home/admin/data/autopirate/bazarr
/home/admin/data/autopirate/sabnzbd_config
/home/admin/data/books/audiobookshelf
/home/admin/data/books/audiobookshelf-metadata
/home/admin/data/books/calibre
/home/admin/data/books/calibre/metadata.db
/home/admin/data/books/calibre-web
/home/admin/data/books/sabnzbd
/home/admin/data/homepage/tasmota/logs
/home/admin/data/jellyfin/jellyfin
/home/admin/data/monitoring/custom.ini
/home/admin/data/monitoring/grafana
/home/admin/data/monitoring/prometheus
/home/admin/data/monitoring/uptime-kuma
/home/admin/data/navidrome
/home/admin/data/plex/ombi
/home/admin/data/plex/plex
/home/admin/data/qbittorrent
/home/admin/data/stash
/home/admin/data/traefik/acme-dns01.json
/home/admin/data/traefik/acme.json
/home/admin/data/unifi_controller/config
/home/admin/data/warpgate
/home/admin/data/wireguard/wg-easy
/home/admin/Jafner.net/homelab/fighter/config/ddns/ddclient
/home/admin/Jafner.net/homelab/fighter/config/homepage/config
/home/admin/Jafner.net/homelab/fighter/config/homepage/icons
/home/admin/Jafner.net/homelab/fighter/config/monitoring/exporter-ping/config.yml
/home/admin/Jafner.net/homelab/fighter/config/monitoring/prometheus.yml
/home/admin/Jafner.net/homelab/fighter/config/traefik/config
/home/admin/Jafner.net/homelab/fighter/config/traefik/traefik.yaml
/home/admin/stacks/gitea-runner/config.yaml
/home/admin/stacks/gitea-runner/registration.token
/home/admin/stacks/home-assistant/mosquitto.conf
/home/admin/stacks/home-assistant/mosquitto.passwd
/var/lib/docker/volumes/026b6b91cdf17575f0948aff512fa381cd39396bf4ecbff4d82cb734567b4e5b/_data
/var/lib/docker/volumes/6216013b5413d62e175974f8f0d923309ea51681f0f7a8205e46fae22f8de8c5/_data
/var/lib/docker/volumes/calibre-web_calibre-web-rpg_data/_data
/var/lib/docker/volumes/calibre-web_calibre-web-sff_data/_data
/var/lib/docker/volumes/cdd5565328632ce00ed158eb1c019f567bcc701588ce583e42efbfc55095ec13/_data
/var/lib/docker/volumes/keycloak_postgres_data/_data
/var/lib/docker/volumes/manyfold_postgres_data/_data
``` /home/admin/data/autopirate/bazarr /home/admin/data/autopirate/sabnzbd_config /home/admin/data/books/audiobookshelf /home/admin/data/books/audiobookshelf-metadata /home/admin/data/books/calibre /home/admin/data/books/calibre/metadata.db /home/admin/data/books/calibre-web /home/admin/data/books/sabnzbd /home/admin/data/homepage/tasmota/logs /home/admin/data/jellyfin/jellyfin /home/admin/data/monitoring/custom.ini /home/admin/data/monitoring/grafana /home/admin/data/monitoring/prometheus /home/admin/data/monitoring/uptime-kuma /home/admin/data/navidrome /home/admin/data/plex/ombi /home/admin/data/plex/plex /home/admin/data/qbittorrent /home/admin/data/stash /home/admin/data/traefik/acme-dns01.json /home/admin/data/traefik/acme.json /home/admin/data/unifi_controller/config /home/admin/data/warpgate /home/admin/data/wireguard/wg-easy /home/admin/Jafner.net/homelab/fighter/config/ddns/ddclient /home/admin/Jafner.net/homelab/fighter/config/homepage/config /home/admin/Jafner.net/homelab/fighter/config/homepage/icons /home/admin/Jafner.net/homelab/fighter/config/monitoring/exporter-ping/config.yml /home/admin/Jafner.net/homelab/fighter/config/monitoring/prometheus.yml /home/admin/Jafner.net/homelab/fighter/config/traefik/config /home/admin/Jafner.net/homelab/fighter/config/traefik/traefik.yaml /home/admin/stacks/gitea-runner/config.yaml /home/admin/stacks/gitea-runner/registration.token /home/admin/stacks/home-assistant/mosquitto.conf /home/admin/stacks/home-assistant/mosquitto.passwd /var/lib/docker/volumes/026b6b91cdf17575f0948aff512fa381cd39396bf4ecbff4d82cb734567b4e5b/_data /var/lib/docker/volumes/6216013b5413d62e175974f8f0d923309ea51681f0f7a8205e46fae22f8de8c5/_data /var/lib/docker/volumes/calibre-web_calibre-web-rpg_data/_data /var/lib/docker/volumes/calibre-web_calibre-web-sff_data/_data /var/lib/docker/volumes/cdd5565328632ce00ed158eb1c019f567bcc701588ce583e42efbfc55095ec13/_data /var/lib/docker/volumes/keycloak_postgres_data/_data /var/lib/docker/volumes/manyfold_postgres_data/_data ```
Jafner commented 2024-10-22 15:03:30 -07:00
Author
Owner
Copy Link

Note: this only includes volumes for services that are running right now. So next we'll look at the dark data we might need to back up.

Note: this only includes volumes for services that are running *right now*. So next we'll look at the dark data we might need to back up.
Jafner commented 2024-10-22 15:12:00 -07:00
Author
Owner
Copy Link

The /home/admin/data directory is only 34.4 GiB. Trivial to back up once we've configured a new storage solution.

The `/home/admin/data` directory is only 34.4 GiB. Trivial to back up once we've configured a new storage solution.
Jafner commented 2024-10-25 21:36:15 -07:00
Author
Owner
Copy Link

Re: Discover solutions for data storage suitable for distributed workloads.

Unless we switch from ZFS to CephFS (and make the massive infra migration necessary to support that), we'll have to stick with old--school SMB and iSCSI for network storage. But we can also bring Longhorn into the equation, which means we'll have three tiers of storage:

  • smb Network file storage
  • iscsi Network block storage
  • longhorn Distributed block storage

This may require adding more physical storage to each node to provide a useful amount storage.

Re: Discover solutions for data storage suitable for distributed workloads. Unless we switch from ZFS to CephFS (and make the massive infra migration necessary to support that), we'll have to stick with old--school SMB and iSCSI for network storage. But we can also bring Longhorn into the equation, which means we'll have three tiers of storage: - `smb` Network file storage - `iscsi` Network block storage - `longhorn` Distributed block storage This may require adding more physical storage to each node to provide a useful amount storage.
Jafner added this to the Ongoing Cleanup and Paying Off Tech Debt milestone 2024-10-25 21:38:47 -07:00
Jafner removed this from the Ongoing Cleanup and Paying Off Tech Debt milestone 2024-10-25 21:38:57 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Jafner/Jafner.net#15
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?