Improve error handling for network devices, clean up echos

docs/Diagram.md

@@ -0,0 +1,34 @@
+```mermaid
+flowchart TD
+    barbarian
+    druid
+    fighter
+    monk
+    wizard
+    cloudflare
+
+cloudflare["Cloudflare DNS"] --DNS *.jafner.tools--> druid["Druid: High-uptime, low data services"]
+cloudflare --DNS *.jafner.net----> wizard["Wizard: Routing with VyOS"]
+wizard --Port forward :80,:443--> fighter["Fighter: Data-dependent services"]
+barbarian["Barbarian: Primary NAS"] --Rsync backup--> monk["Monk: Backup NAS"]
+
+druid --Docker--> 5eTools["5eTools: D&D 5th Edition Wiki"]
+druid --Docker--> Gitea["Gitea: This Git server!"]
+druid --Docker--> Uptime-Kuma["Uptime-Kuma: Synthetic monitoring and alerting"]
+druid --Docker--> Vaultwarden["Vaultwarden: Self-hosted Bitwarden server"]
+druid --Docker--> Wireguard["Wireguard: Performant VPN"]
+
+fighter --Docker--> Autopirate["Autopirate: Stack of applications for downloading Linux ISOs"] <--SMB--> barbarian
+fighter --Docker--> Calibre-web["Calibre-web: Ebook library frontend"] <--SMB--> barbarian
+fighter --Docker--> Keycloak["Keycloak: SSO Provider"] 
+fighter --Docker--> Minecraft["Minecraft Servers"] <--iSCSI--> barbarian
+fighter --Docker--> Grafana["Grafana, Prometheus, Uptime-Kuma"]
+fighter --Docker--> Nextcloud["Nextcloud: Cloud drive and office suite"] <--iSCSI--> barbarian
+fighter --Docker--> Plex["Plex: Media library frontend"] <--SMB--> barbarian
+fighter --Docker--> Qbittorrent["Qbittorrent: Torrent client"] <--SMB--> barbarian
+fighter --Docker--> Send["Send: Self-hosted Firefox Send"] <--iSCSI--> barbarian
+fighter --Docker--> Stash["Stash: Linux ISO frontend"] <--SMB--> barbarian
+fighter --Docker--> Unifi["Unifi controller"]
+fighter --Docker--> Vandam["Manyfold: 3D Asset library manager"] <--SMB--> barbarian
+fighter --Docker--> Wireguard2["Wireguard: Performant VPN"]
+```

docs/iSCSI.md

@@ -1,3 +1,11 @@
+# NOTE: Adding or removing drives
+> The drive letter of the iSCSI device will change (e.g. from `/dev/sde` to `/dev/sdb`) if drives are added or removed. This will cause the mount to fail.  
+
+To resolve:
+0. Make sure all Docker stacks relying on the iSCSI drive are shut down. 
+1. Update the `fstab` entry. Edit the `/etc/fstab` file as root, and update the drive letter. 
+2. Re-mount the drive. Run `sudo mount -a`. 
+
 # Creating the Zvol and iSCSI share in TrueNAS Scale
 
 1. Navigate to the dataset to use. From the TrueNAS Scale dashboard, open the navigation side panel. Navigate to "Datasets". Select the pool to use (`Tank`).  
@@ -16,16 +24,16 @@
 2. Get the list of available shares from the NAS with `sudo iscsiadm --mode discovery --type sendtargets --portal 192.168.1.10` where the IP for `--portal` is the IP of the NAS hosting the iSCSI share. In my case, this returns `192.168.1.10:3260,1 iqn.2020-03.net.jafner:fighter`. 
 3. Open the iSCSI session. Run `sudo iscsiadm --mode node --targetname "iqn.2020-03.net.jafner:fighter" --portal "192.168.1.10:3260" --login`. Where the name for `--targetname` is the iqn string including the share name. And where the address for `--portal` has both the IP and port used by the NAS hosting the iSCSI share. Verify the session connected with `sudo iscsiadm --mode session --print=1`, which should return the description of any active sessions. [Debian.org](https://wiki.debian.org/SAN/iSCSI/open-iscsi).
 4. Format the newly-added block device. 
-  a. Identify the device name of the new device with `sudo iscsiadm -m session -P 3 | grep "Attached scsi disk"`. In my case, `sde`. [ServerFault](https://serverfault.com/questions/828401/how-can-i-determine-if-an-iscsi-device-is-a-mounted-linux-filesystem).
-  b. Partition and format the device. Run `sudo parted --script /dev/sde "mklabel gpt" && sudo parted --script /dev/sde "mkpart primary 0% 100%" && sudo mkfs.ext4 /dev/sde1` [Server-world.info](https://www.server-world.info/en/note?os=Debian_11&p=iscsi&f=3). 
-  c. Mount the new partition to a directory. Run `sudo mkdir /mnt/iscsi && sudo mount /dev/sde1 /mnt/iscsi`. Where the path `/dev/sde1` is the newly-created partition and the path `/mnt/iscsi` is the path to which you want it mounted.
+  a. Identify the device name of the new device with `sudo iscsiadm -m session -P 3 | grep "Attached scsi disk"`. In my case, `sdb`. [ServerFault](https://serverfault.com/questions/828401/how-can-i-determine-if-an-iscsi-device-is-a-mounted-linux-filesystem).
+  b. Partition and format the device. Run `sudo parted --script /dev/sdb "mklabel gpt" && sudo parted --script /dev/sdb "mkpart primary 0% 100%" && sudo mkfs.ext4 /dev/sdb1` [Server-world.info](https://www.server-world.info/en/note?os=Debian_11&p=iscsi&f=3). 
+  c. Mount the new partition to a directory. Run `sudo mkdir /mnt/iscsi && sudo mount /dev/sdb1 /mnt/iscsi`. Where the path `/dev/sdb1` is the newly-created partition and the path `/mnt/iscsi` is the path to which you want it mounted.
   d. Test the disk write speed of the new partition. Run `sudo dd if=/dev/zero of=/mnt/iscsi/temp.tmp bs=1M count=32768` to run a 32GB test write. [Cloudzy.com](https://cloudzy.com/blog/test-disk-speed-in-linux/).
 
 # Connecting and mounting the iSCSI share on boot
 
 1. Get the full path of the share's configuration. It should be like `/etc/iscsi/nodes/<share iqn>/<share host address>/default`. In my case it was `/etc/iscsi/nodes/iqn.202-03.net.jafner:fighter/192.168.1.10,3260,1/default`. [Debian.org](https://wiki.debian.org/SAN/iSCSI/open-iscsi). 
 2. Set the `node.startup` parameter to `automatic`. Run `sudo sed -i 's/node.startup = manual/node.startup = automatic/g' /etc/iscsi/nodes/iqn.2020-03.net.jafner:fighter/192.168.1.10,3260,1/default`. 
-3. Add the new mount to `/etc/fstab`. Run `sudo bash -c "echo '/dev/sde1 /mnt/iscsi ext4 _netdev 0 0' >> /etc/fstab"`. [Adamsdesk.com](https://www.adamsdesk.com/posts/sudo-echo-permission-denied/), [StackExchange](https://unix.stackexchange.com/questions/195116/mount-iscsi-drive-at-boot-system-halts). 
+3. Add the new mount to `/etc/fstab`. Run `sudo bash -c "echo '/dev/sdb1 /mnt/iscsi ext4 _netdev 0 0' >> /etc/fstab"`. [Adamsdesk.com](https://www.adamsdesk.com/posts/sudo-echo-permission-denied/), [StackExchange](https://unix.stackexchange.com/questions/195116/mount-iscsi-drive-at-boot-system-halts). 
 
 # How to Gracefully Terminate iSCSI Session
 

fighter/scripts/startup.sh

@@ -3,25 +3,32 @@
 # Check for network mounted devices
 # NAS SMB
 if ! mount -t cifs | grep -q '/mnt/nas'; then
-    echo "NAS SMB shares not mounted"
+    echo "  ==== NAS SMB shares not mounted"
     exit 1
 fi
 
 # NAS iSCSI
 if ! sudo iscsiadm -m session | grep -q 'iqn.2020-03.net.jafner:fighter'; then 
-    echo "NAS iSCSI share not mounted"
-    exit 1
+    echo "  ==== NAS iSCSI session not connected"
+    if ! mount -t ext4 | grep -q '/mnt/iscsi'; then
+        echo "  ==== /mnt/iscsi not mounted"
+        exit 1
+    fi
 fi
 
 for stack in /home/admin/homelab/fighter/config/*; do
     cd $stack 
+    if ! docker compose config; then 
+        echo "  ==== Invalid compose config: $stack"
+    fi
+    echo "  ==== Bringing up $stack"
     docker compose up -d 
     cd /home/admin/homelab/fighter/config/
 done
 
 # extra thing because my keycloak healthcheck doesn't work properly
 
-echo "Wait 15s, then bring Keycloak forwardauth containers online"
+echo "  ==== Wait 15s, then bring Keycloak forwardauth containers online"
 cd /home/admin/homelab/fighter/config/keycloak
 sleep 15
 docker compose up -d