homelab

Monolithic repository for my homelab. Everything described below is updated manually. So it's probably a little out of date. Or a lot out of date.

The Metal

This lab was cobbled over time, as many are, from a combination of three sources:

• Hardware handed down from my gaming PC.
• Ebay used enterprise hardware.
• Cheap new stuff that you don't want to skimp on (PSUs).

Most of each host's hardware configuration can be found in an inxi.txt file. For example, fighter/inxi.txt. These are simply copied from the output of inxi -b.

The Compute Platform

On top of each host's hardware, we install a Linux, Debian-based operating system. I've picked TrueNAS Scale, VyOS, and Debian itself to preserve some consistency in tooling between hosts. But at the same time benefitting from application-specific distributions.

In the case of our Debian-based systems, we install the Docker engine to run our applications.

Secrets and Security

Our repository contains as many configuration details as reasonable. But we must secure our secrets: passwords, API keys, encryption seeds, etc..

Docker Env Vars

1. We store our Docker env vars in a file named after the service. For example keycloak.env.
2. We separate our secrets from non-secret env vars by placing them in a file with a similar name, but with _secrets appended to the service name. For example keycloak_secrets.env. These files exist only on the host for which they are necessary, and must be created manually on the host.
3. Our repository .gitignore excludes all files matching *.secret, and *_secrets.env.

Note: This makes secrets very fragile. Accidental deletion or other data loss can destroy the secret permanently.

Generating Secrets

We use the password manager's generator to create secrets with the desired parameters, preferring the following parameters: - 64 characters - Capital letters, lowercase letters, numbers, and standard symbols (^*@#!&$%) If necessary, we will reduce characterset by cutting out symbols before reducing string length.

Host OS Initial Setup

For general-purpose hosts, we start from an up-to-date Debian base image. For appliances and application-specific hosts, we prefer downstream of Debian for consistency.

General Purpose Packages

Assuming a Debian base image, we install the following basic packages:

1. curl to facilitate web requests for debugging.
2. nano as preferred terminal text editor.
3. inxi to compile hardware info.
4. git to interact with homelab config repo.
5. htop to view primary host resources in real time.

Installing Docker

There are two modes of running Docker: root and rootless. Docker was built to run as root, and running as root is much more convenient. However, any potential vulnerabilities in Docker risk privilege escalation.

Installing Docker in Root mode (current, deprecated)

We use the convenient, insecure install script to install docker.

1. curl -fsSL https://get.docker.com | sudo sh to get and run the install script.
2. sudo systemctl enable docker to enable the Docker daemon service.
3. sudo usermod -aG docker $USER to add the current user (should be "admin") to the docker group.
4. logout to log out as the current user. Log back in to apply new perms.
5. docker ps should now return an empty table.

https://docs.docker.com/engine/install/debian/

Installing Docker in Rootless mode (preferred)

This is the preferred process, as rootless mode mitigates many potential vulnerabilities in the Docker application and daemon.

1. sudo apt-get update && sudo apt-get install uidmap dbus-user-session fuse-overlayfs slirp4netns to install the prerequisite packages to enable rootless mode.
2. Set up the Docker repository:

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

3. Install the Docker packages:

sudo apt-get install \
  docker-ce \
  docker-ce-cli \
  containerd.io \
  docker-buildx-plugin \
  docker-compose-plugin \
  docker-ce-rootless-extras

4. Run the rootless setup script with dockerd-rootless-setuptool.sh install
5. systemctl --user start docker to start the rootless docker daemon.
6. systemctl --user enable docker && sudo loginctl enable-linger $(whoami) to configure the rootless docker daemon to run at startup.
7. export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock && docker context use rootless to configure the client to connect to the socket.

Theoretically, this should work according to the Docker docs. But when I attempted to follow these steps I got the following error when attempting to create a basic nginx container:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-1c7f642e0716cf1a67c6a0c6ad4a1de3833eb82682ce62b219f423fa1014e227.scope" (properties [{Name:Description Value:"libcontainer container 1c7f642e0716cf1a67c6a0c6ad4a1de3833eb82682ce62b219f423fa1014e227"} {Name:Slice Value:"user.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [39360]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Permission denied: unknown.

https://docs.docker.com/engine/security/rootless/

Linux User Management

We create a non-root user (usually called "admin") with a strong password and passwordless sudo.

On Debian-based systems, we take the following steps:

1. As root user, run adduser admin to create the non-root user called "admin".
2. As root user, run usermod -aG sudo admin to add the new "admin" user to the sudo group.
3. As root user, run visudo and append this line to the end of the file: admin ALL=(ALL) NOPASSWD:ALL.
4. Switch to the new user with sudo su admin.
5. As the new "admin" user, run passwd to create a new, strong password. Generate this password with the password manager and store it under the SSH Hosts folder.

https://www.cyberciti.biz/faq/add-new-user-account-with-admin-access-on-linux/ https://www.cyberciti.biz/faq/linux-unix-running-sudo-command-without-a-password/ https://www.cyberciti.biz/faq/linux-set-change-password-how-to/

Securing SSH

For all hosts we want to take the standard steps to secure SSH access.

1. mkdir /home/$USER/.ssh to create the ~/.ssh directory for the non-root user (usually "admin").
2. Copy your SSH public key to the clipboard, then echo "<insert pubkey here>" >> /home/admin/.ssh/authorized_keys to enable key-based SSH access to the user.
3. Install the authenticator libpam plugin package with sudo apt install libpam-google-authenticator
4. Run the authenticator setup with google-authenticator and use the following responses:
   • Do you want authentication tokens to be time-based? y
   • Do you want me to update your "/home/$USER/.google_authenticator" file? y
   • Do you want to disallow multiple uses of the same authentication token? y
   • Do you want to do so? n (refers to increasing time skew window)
   • Do you want to enable rate-limiting? y We enter our TOTP secret key into our second authentication method and save our one-time backup recovery codes.
5. Edit the /etc/pam.d/sshd file as sudo, and add this line to the top of the file auth sufficient pam_google_authenticator.so nullok.
6. Edit the /etc/ssh/sshd_config file as sudo, and ensure the following assertions exist:
   • PubkeyAuthentication yes to enable authentication via pubkeys in ~/.ssh/authorized_keys.
   • AuthenticationMethods publickey,keyboard-interactive to allow both pubkey and the interactive 2FA prompt.
   • PasswordAuthentication no to disable password-based authentication.
   • ChallengeResponseAuthentication yes to enable 2FA interactive challenge.
   • UsePAM yes to use the 2FA authenticator libpam module.
7. Restart the SSH daemon with sudo systemctl restart sshd.service.

Note: SSH root login will be disabled implicitly by requiring pubkey authentication and having no pubkeys listed in /root/.ssh/authorized_keys.

https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04

Disabling 2FA

Some use cases (such as programmatic access) demand 2FA be disabled. Some day we'll figure out how to allow specific keys to bypass the 2FA requirement. But until then,

1. Edit the file /etc/pam.d/sshd and comment out the line auth sufficient pam_google_authenticator.so nullok
2. Edit the file /etc/ssh/sshd_config and find the AuthenticationMethods configuration. Replace the value publickey,keyboard-interactive with publickey.

SSH Key Management

The process for managing SSH keys should work as follows:

1. SSH access to hosts should be controlled via keys listed in ~/.ssh/authorized_keys.
2. One key should map to one user on one device.
3. When authorizing a key, review existing authorized keys and remove as appropriate.
4. Device keys should be stored under the "SSH Keys" folder in the password manager. The pubkey should be the "password" for easy copying, and the private component should be added as an attachment.

Patching and Updating

In the interest of proactively mitigating security risks, we try to keep packages up to date. We have two main concerns for patching: host packages, and docker images. Each of these have their own concerns and are handled separately.

Host Packages via Unattended Upgrades

Since Debiant 9, the unattended-upgrades and apt-listchanges are installed by default.

1. Install the packages with sudo apt-get install unattended-upgrades apt-listchanges.
2. Create the default automatic upgrade config with sudo dpkg-reconfigure -plow unattended-upgrades

By default, we will get automatic upgrades for the distro version default and security channels (e.g. bullseye and bullseye-security) with the Debian and Debian-Security labels.

https://wiki.debian.org/UnattendedUpgrades

Debian Version Upgrade

When the time comes for a major version upgrade on a Debian system, we take the following steps as soon as realistic.

1. Update the current system with sudo apt-get update && sudo apt-get upgrade && sudo apt-get full-upgrade.
2. Switch the update channel for APT sources. 2a. Export the name of the new version codename to a variable with NEW_VERSION_CODENAME=bookworm (bookworm as an example). 2b. for file in /etc/apt/sources.list /etc/apt/sources.list.d/*; do sudo sed "s/$VERSION_CODENAME/$NEW_VERSION_CODENAME/g" $file; done.
3. Clean out old packages and pull the new lists sudo apt-get clean && sudo apt-get update
4. Update to most recent versions of all packages for new channel with sudo apt-get upgrade && sudo apt-get full-upgrade
5. Clean out unnecessary packages with sudo apt-get autoremove.
6. Reboot the host to finalize changes with sudo shutdown -r now.

Note: If migrating from Debian versions <12 to versions >=12, add the following repos (in addition to main) after step 2a: contrib non-free non-free-firmware.

https://wiki.debian.org/DebianUpgrade

Docker Images

As of now, we have no automated process or tooling for updating Docker images.

We usually update Docker images one stack at a time. For example, we'll update calibre-web on fighter:

1. Navigate to the directory of the stack. cd ~/homelab/fighter/config/calibre-web
2. Check the images and tags to be pulled with docker-compose config | grep image
3. Pull the latest version of the image tagged in the compose file docker-compose pull
4. Restart the containers to use the new images with docker-compose up -d --force-recreate

Note: We can update one image from a stack by specifying the name of the service. E.g. docker-compose pull forwardauth