Move Docker data from server to NAS #72

Closed
opened 2022-11-27 22:01:53 -08:00 by Jafner · 30 comments
Jafner commented 2022-11-27 22:01:53 -08:00 (Migrated from gitlab.jafner.net)
  • Create and mount NAS share (`/Tank/Docker Data/')
  • Inventory services to be migrated
  • Offline all services to be migrated
  • Copy data from server to share
  • Reconfigure service volumes
  • Online services with new configuration
- [X] Create and mount NAS share (`/Tank/Docker Data/') - [x] Inventory services to be migrated - [x] Offline all services to be migrated - [x] Copy data from server to share - [x] Reconfigure service volumes - [x] Online services with new configuration
Jafner commented 2022-11-27 22:01:53 -08:00 (Migrated from gitlab.jafner.net)

assigned to @Jafner

assigned to @Jafner
Jafner commented 2022-11-27 22:02:02 -08:00 (Migrated from gitlab.jafner.net)

changed the description

changed the description
Jafner commented 2022-11-27 22:22:59 -08:00 (Migrated from gitlab.jafner.net)

changed the description

changed the description
Jafner commented 2022-11-27 22:29:08 -08:00 (Migrated from gitlab.jafner.net)

Re: "Inventory services to be migrated"
Below are listed services that store data at /home/joey/data/$app. Bolded services should be migrated.

  • Autopirate
  • Fireshare
  • Gitlab
  • Jdownloader
  • Keycloak
  • Matrix
  • Megan-portfolio
  • Monitoring
  • Nextcloud
  • Plex
  • Qbittorrent
  • Send
  • SFTP
  • Stash
  • Traefik
  • Unifi_controller
  • Wireguard
Re: "Inventory services to be migrated" Below are listed services that store data at `/home/joey/data/$app`. Bolded services should be migrated. - **Autopirate** - Fireshare - Gitlab - **Jdownloader** - Keycloak - Matrix - Megan-portfolio - Monitoring - **Nextcloud** - **Plex** - Qbittorrent - **Send** - **SFTP** - Stash - Traefik - Unifi_controller - Wireguard
Jafner commented 2022-11-27 22:29:12 -08:00 (Migrated from gitlab.jafner.net)

marked the checklist item Inventory services to be migrated as completed

marked the checklist item **Inventory services to be migrated** as completed
Jafner commented 2022-11-27 22:53:03 -08:00 (Migrated from gitlab.jafner.net)

Re: "Offline all services to be migrated",
Re: "Copy data from server to share",
Re: "Reconfigure service volumes"

  • Autopirate
  • Jdownloader
  • Nextcloud
  • Plex
  • Send
  • SFTP
Re: "Offline all services to be migrated", Re: "Copy data from server to share", Re: "Reconfigure service volumes" - [x] Autopirate - [x] Jdownloader - [x] Nextcloud - [x] Plex - [x] Send - [ ] ~~SFTP~~
Jafner commented 2022-11-27 23:20:24 -08:00 (Migrated from gitlab.jafner.net)

We're seeing extremely slow copy speeds via cp. This must be diagnosed before we can rely on this network share for docker data.

image

We're seeing extremely slow copy speeds via `cp`. This must be diagnosed before we can rely on this network share for docker data. ![image](/uploads/9296a16d0e78a4e261a0dd5f7ca854cb/image.png)
Jafner commented 2022-11-27 23:29:29 -08:00 (Migrated from gitlab.jafner.net)

The directory in question contains 297,012 files totalling only 18 Gigabytes. The slow copy speed may be due to the many small files (average 65KB).

The directory in question contains 297,012 files totalling only 18 Gigabytes. The slow copy speed may be due to the many small files (average 65KB).
Jafner commented 2022-11-27 23:35:05 -08:00 (Migrated from gitlab.jafner.net)

Used sudo iotop to get disk IO monitor.
Used du -sh . to get directory size.
Used find autopirate/ -type f | wc -l to get file count.

Used `sudo iotop` to get disk IO monitor. Used `du -sh .` to get directory size. Used `find autopirate/ -type f | wc -l` to get file count.
Jafner commented 2022-11-27 23:41:20 -08:00 (Migrated from gitlab.jafner.net)

During the same operation we're seeing much faster copy speeds. Though it does not appear to be consistent, we can assume this speed is a function of the files being copied.

image

During the same operation we're seeing much faster copy speeds. Though it does not appear to be consistent, we can assume this speed is a function of the files being copied. ![image](/uploads/b304237590f4fb32315c4133b64a3321/image.png)
Jafner commented 2022-11-27 23:41:57 -08:00 (Migrated from gitlab.jafner.net)

image

![image](/uploads/b04d34c6849e9722d440cc5ab11ac969/image.png)
Jafner commented 2022-11-27 23:53:03 -08:00 (Migrated from gitlab.jafner.net)

It appears using rsync -avzh provides consistently better performance.

image

It appears using `rsync -avzh` provides consistently better performance. ![image](/uploads/8a3f29ebc70f94e81cb1dc05ee7c6684/image.png)
Jafner commented 2022-11-28 00:09:48 -08:00 (Migrated from gitlab.jafner.net)

We don't need to migrate SFTP. It has never been used, and is unlikely to ever be used.

We don't need to migrate SFTP. It has never been used, and is unlikely to ever be used.
Jafner commented 2022-11-28 00:17:15 -08:00 (Migrated from gitlab.jafner.net)

mentioned in commit e59148bccc

mentioned in commit e59148bccc18a6a15f5d66bd5b7c89dd69d7c7b2
Jafner commented 2022-11-28 00:52:38 -08:00 (Migrated from gitlab.jafner.net)

mentioned in commit 52862aaf1d

mentioned in commit 52862aaf1d9885f6cc4dd5cd101aca2ed54115b4
Jafner commented 2022-11-28 09:24:01 -08:00 (Migrated from gitlab.jafner.net)

This might be a real problem.
Permissions for files on the network.

image

This might be a real problem. Permissions for files on the network. ![image](/uploads/69682a5c899c37d999d83e23bf95ca0c/image.png)
Jafner commented 2022-11-28 09:28:48 -08:00 (Migrated from gitlab.jafner.net)

for app in autopirate jdownloader2 nextcloud plex send; do cd $app && docker-compose down && cd ~/homelab/server/config; done

`for app in autopirate jdownloader2 nextcloud plex send; do cd $app && docker-compose down && cd ~/homelab/server/config; done`
Jafner commented 2022-11-28 09:38:01 -08:00 (Migrated from gitlab.jafner.net)

While we work out a permissions-sensitive solution to NAS-based file storage, we're going to use /mnt/md0 for permissions-sensitive workloads. For now that just means Nextcloud and Ombi.

While we work out a permissions-sensitive solution to NAS-based file storage, we're going to use `/mnt/md0` for permissions-sensitive workloads. For now that just means Nextcloud and Ombi.
Jafner commented 2022-11-28 09:40:56 -08:00 (Migrated from gitlab.jafner.net)

mentioned in commit 4c74ca407a

mentioned in commit 4c74ca407afa1e352b5c4e48d736f37369bf3b78
Jafner commented 2022-11-28 09:46:03 -08:00 (Migrated from gitlab.jafner.net)

It seems Ombi is experiencing a similar issue:

image

It seems [Ombi](https://ombi.jafner.net) is experiencing a similar issue: ![image](/uploads/1170d83be6124edd15a84e248e1f23f2/image.png)
Jafner commented 2022-11-28 09:48:19 -08:00 (Migrated from gitlab.jafner.net)

We're gonna move Plex back over to the local drive. (~/data/plex)

We're gonna move Plex back over to the local drive. (`~/data/plex`)
Jafner commented 2022-11-28 09:48:47 -08:00 (Migrated from gitlab.jafner.net)

mentioned in commit b112540604

mentioned in commit b112540604f2d7948807dd5500623773de3f330c
Jafner commented 2022-11-28 09:49:26 -08:00 (Migrated from gitlab.jafner.net)

This is confirmed to fix the Ombi issue.

This is confirmed to fix the Ombi issue.
Jafner commented 2022-11-28 09:59:20 -08:00 (Migrated from gitlab.jafner.net)

Re: "Online services with new configuration"

  • Autopirate
  • Jdownloader
  • Nextcloud
  • Plex
  • Send
Re: "Online services with new configuration" - [X] Autopirate - [X] Jdownloader - [X] Nextcloud - [X] Plex - [X] Send
Jafner commented 2022-11-28 11:27:14 -08:00 (Migrated from gitlab.jafner.net)

marked the checklist item Offline all services to be migrated as completed

marked the checklist item **Offline all services to be migrated** as completed
Jafner commented 2022-11-28 11:27:16 -08:00 (Migrated from gitlab.jafner.net)

marked the checklist item Copy data from server to share as completed

marked the checklist item **Copy data from server to share** as completed
Jafner commented 2022-11-28 11:27:17 -08:00 (Migrated from gitlab.jafner.net)

marked the checklist item Reconfigure service volumes as completed

marked the checklist item **Reconfigure service volumes** as completed
Jafner commented 2022-11-28 11:27:19 -08:00 (Migrated from gitlab.jafner.net)

marked the checklist item Online services with new configuration as completed

marked the checklist item **Online services with new configuration** as completed
Jafner commented 2022-11-28 11:27:51 -08:00 (Migrated from gitlab.jafner.net)

We're going to close this ticket and open another for working out a long-term solution for Nextcloud storage.

We're going to close this ticket and open another for working out a long-term solution for Nextcloud storage.
Jafner commented 2022-11-28 11:30:59 -08:00 (Migrated from gitlab.jafner.net)

mentioned in issue #73

mentioned in issue #73
Sign in to join this conversation.
No Label
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Jafner/homelab#72
No description provided.