From 36c6c0be75ce8bbb18685954ea2080226c3d7d4c Mon Sep 17 00:00:00 2001 From: Joey Hafner Date: Sat, 17 Feb 2024 19:39:06 -0800 Subject: [PATCH] #112 Init reorganized README --- README.md | 260 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 193 insertions(+), 67 deletions(-) diff --git a/README.md b/README.md index d8637fb..235a827 100644 --- a/README.md +++ b/README.md @@ -1,77 +1,203 @@ # homelab -Monolithic repository for my homelab +Monolithic repository for my homelab. +Everything described below is updated manually. So it's probably a little out of date. Or a lot out of date. -# Navigation -This repo is (mostly) organized into the following structure: -```bash -/ -# The root contains repository meta-information like .gitignore, -# .gitlab-ci.yml, .gitmodules, and README.md. - docs/ - # The /docs directory is for all self-contained documentation - # that is not tied to a specific service. Service-specific documentation - # is contained in /$host/config/$service/README.md - img/ - # supporting images for use in docs +# The Metal +This lab was cobbled over time, as many are, from a combination of three sources: - $host/ - # There are separate directories for the details and configuration of - # each host. At the root of `/$host/` we have non-authoritative - # documentation and reference. This includes printouts of hardware - # configs (`inxi -b`), host-specific procedure docs, useful scripts, etc. - scripts/ - # if a host has scripts for automating recurring tasks, - # they will be placed here. - config/ - # Anything in the `/$host/config` directory is used as a source of - # truth from which hosts pull and apply the defined configuration. - $service/ - # for Docker-enabled hosts each service stack will be - # configured within a directory - docker-compose.yml - # all services (except minecraft, which needed a - # more modular system) use docker-compose.yml to - # define their stack configuration. - .env - # contains volume mapping env vars, used automatically by - # docker-compose to expand ${...}. - README.md - # if a service stack has documentation specific to itself, - # it will be contained within this file. This usually contains - # procedure for interacting with a container and system configuration - # changes that could not be tracked in code (e.g. /etc/fstab or - # crontab or /etc/docker/daemon.json) +- Hardware handed down from my gaming PC. +- Ebay used enterprise hardware. +- Cheap new stuff that you don't want to skimp on (PSUs). + +Most of each host's hardware configuration can be found in an `inxi.txt` file. For example, [`fighter/inxi.txt`](fighter/inxi.txt). These are simply copied from the output of `inxi -b`. + +# The Compute Platform +On top of each host's hardware, we install a Linux, Debian-based operating system. I've picked TrueNAS Scale, VyOS, and Debian itself to preserve some consistency in tooling between hosts. But at the same time benefitting from application-specific distributions. + +In the case of our Debian-based systems, we install the Docker engine to run our applications. + +# Secrets and Security +Our repository contains as many configuration details as reasonable. But we must secure our secrets: passwords, API keys, encryption seeds, etc.. + +## Docker Env Vars +1. We store our Docker env vars in a file named after the service. For example `keycloak.env`. +2. We separate our secrets from non-secret env vars by placing them in a file with a similar name, but with `_secrets` appended to the service name. For example `keycloak_secrets.env`. These files exist only on the host for which they are necessary, and must be created manually on the host. +3. Our repository `.gitignore` excludes all files matching `*.secret`, and `*_secrets.env`. + +Note: This makes secrets very fragile. Accidental deletion or other data loss can destroy the secret permanently. + +## Generating Secrets +We use the password manager's generator to create secrets with the desired parameters, preferring the following parameters: + - 64 characters + - Capital letters, lowercase letters, numbers, and standard symbols (`^*@#!&$%`) +If necessary, we will reduce characterset by cutting out symbols before reducing string length. + +## Host OS Initial Setup +For general-purpose hosts, we start from an up-to-date Debian base image. For appliances and application-specific hosts, we prefer downstream of Debian for consistency. + +### General Purpose Packages +Assuming a Debian base image, we install the following basic packages: + +1. `curl` to facilitate web requests for debugging. +2. `nano` as preferred terminal text editor. +3. `inxi` to compile hardware info. +4. `git` to interact with homelab config repo. +5. `htop` to view primary host resources in real time. + +### Installing Docker +There are two modes of running Docker: root and rootless. +Docker was built to run as root, and running as root is much more convenient. However, any potential vulnerabilities in Docker risk privilege escalation. + +#### Installing Docker in Root mode (current, deprecated) +We use the convenient, insecure install script to install docker. +1. `curl -fsSL https://get.docker.com | sudo sh` to get and run the install script. +2. `sudo systemctl enable docker` to enable the Docker daemon service. +3. `sudo usermod -aG docker $USER` to add the current user (should be "admin") to the docker group. +4. `logout` to log out as the current user. Log back in to apply new perms. +5. `docker ps` should now return an empty table. + +https://docs.docker.com/engine/install/debian/ + +#### Installing Docker in Rootless mode (preferred) +This is the preferred process, as rootless mode mitigates many potential vulnerabilities in the Docker application and daemon. + +1. `sudo apt-get update && sudo apt-get install uidmap dbus-user-session fuse-overlayfs slirp4netns` to install the prerequisite packages to enable rootless mode. +2. Set up the Docker repository: + +```sh +sudo apt-get update +sudo apt-get install ca-certificates curl gnupg +sudo install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg +sudo chmod a+r /etc/apt/keyrings/docker.gpg + +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \ + $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ + sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update ``` -# Setting Up the Repository -1. Create a new Gitlab [Personal Access Token](https://gitlab.jafner.net/-/profile/personal_access_tokens) named after the host on which it will be used. It should have the scopes `read_api`, `read_user`, `read_repository`, and, optionally, `write_repository` if the host will be pushing commits back to the origin. Development hosts should have the `write_repository` permission. Note the *token name* and *token key* for step 6. -2. `mkdir ~/homelab ~/data && cd ~/homelab` Create the `~/homelab` and `~/data` directories. This should be under the `admin` user's home directory, or equivalent. *It should not be owned by root.* -3. `git init` Initialize the git repo. It should be empty at this point. We must init the repo empty in order to configure sparse checkout. -4. `git config core.sparseCheckout true && git config core.fileMode false && git config pull.ff only && git config init.defaultBranch main` Configure the repo to use sparse checkout and ignore file mode changes. Also configure default branch and pull behavior. -5. (Optional) `echo "$HOSTNAME/" > .git/info/sparse-checkout` Configure the repo to checkout only the files relevant to the host (e.g. fighter). Development hosts should not use this. -6. `git remote add -f origin https://:@gitlab.jafner.net/Jafner/homelab.git` Add the origin with authentication via personal access token and fetch. Remember to replace the placeholder token name and token key with the values from step 1. -7. `git checkout main` Checkout the main branch to fetch the latest files. +3. Install the Docker packages: -## Disabling Sparse Checkout -To disable sparse checkout, simply run `git sparse-checkout disable`. -With this, it can also be re-eneabled with `git sparse-checkout init`. -You can use these two commands to toggle sparse checkout. -Per: https://stackoverflow.com/questions/36190800/how-to-disable-sparse-checkout-after-enabled - -# Debian: Setting FQDN Hostname PS1 -By default, Debian will print the domain part of its hostname (e.g. hostname `jafner.net` will print `jafner` in the terminal prompt). I like to separate my hosts by TLD (e.g. `jafner.net`, `jafner.chat`, `jafner.tools`, etc.), so printing the TLD in the hostname for the PS1 is useful. - -For a standard Debian 11 installation, the PS1 is set in this block of `~/.bashrc`: - -```bash -if [ "$color_prompt" = yes ]; then - PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' -else - PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ ' -fi +```sh +sudo apt-get install \ + docker-ce \ + docker-ce-cli \ + containerd.io \ + docker-buildx-plugin \ + docker-compose-plugin \ + docker-ce-rootless-extras ``` -We can achieve the desired behavior by replacing instances of `\h` with `\H` in both of these prompts (and elsewhere in the bashrc file if necessary). Note: Don't forget to `source ~/.bashrc` to apply the new configuration! +4. Run the rootless setup script with `dockerd-rootless-setuptool.sh install` +5. `systemctl --user start docker` to start the rootless docker daemon. +6. `systemctl --user enable docker && sudo loginctl enable-linger $(whoami)` to configure the rootless docker daemon to run at startup. +7. `export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock && docker context use rootless` to configure the client to connect to the socket. -Reference: [Cyberciti.biz - How to Change / Set up bash custom prompt (PS1) in Linux](https://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html) \ No newline at end of file +Theoretically, this should work according to the Docker docs. But when I attempted to follow these steps I got the following error when attempting to create a basic nginx container: + +``` +docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit "docker-1c7f642e0716cf1a67c6a0c6ad4a1de3833eb82682ce62b219f423fa1014e227.scope" (properties [{Name:Description Value:"libcontainer container 1c7f642e0716cf1a67c6a0c6ad4a1de3833eb82682ce62b219f423fa1014e227"} {Name:Slice Value:"user.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [39360]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Permission denied: unknown. +``` + +https://docs.docker.com/engine/security/rootless/ + +## Linux User Management +We create a non-root user (usually called "admin") with a strong password and passwordless sudo. + +On Debian-based systems, we take the following steps: + +1. As root user, run `adduser admin` to create the non-root user called "admin". +2. As root user, run `usermod -aG sudo admin` to add the new "admin" user to the sudo group. +3. As root user, run `visudo` and append this line to the end of the file: `admin ALL=(ALL) NOPASSWD:ALL`. +4. Switch to the new user with `sudo su admin`. +5. As the new "admin" user, run `passwd` to create a new, strong password. Generate this password with the password manager and store it under the SSH Hosts folder. + +https://www.cyberciti.biz/faq/add-new-user-account-with-admin-access-on-linux/ +https://www.cyberciti.biz/faq/linux-unix-running-sudo-command-without-a-password/ +https://www.cyberciti.biz/faq/linux-set-change-password-how-to/ + + +## Securing SSH + +For all hosts we want to take the standard steps to secure SSH access. + +1. `mkdir /home/$USER/.ssh` to create the `~/.ssh` directory for the non-root user (usually "admin"). +2. Copy your SSH public key to the clipboard, then `echo "" >> /home/admin/.ssh/authorized_keys` to enable key-based SSH access to the user. +3. Install the authenticator libpam plugin package with `sudo apt install libpam-google-authenticator` +4. Run the authenticator setup with `google-authenticator` and use the following responses: + - Do you want authentication tokens to be time-based? `y` + - Do you want me to update your "/home/$USER/.google_authenticator" file? `y` + - Do you want to disallow multiple uses of the same authentication token? `y` + - Do you want to do so? `n` (refers to increasing time skew window) + - Do you want to enable rate-limiting? `y` We enter our TOTP secret key into our second authentication method and save our one-time backup recovery codes. +5. Edit the `/etc/pam.d/sshd` file as sudo, and add this line to the top of the file `auth sufficient pam_google_authenticator.so nullok`. +6. Edit the `/etc/ssh/sshd_config` file as sudo, and ensure the following assertions exist: + - `PubkeyAuthentication yes` to enable authentication via pubkeys in `~/.ssh/authorized_keys`. + - `AuthenticationMethods publickey,keyboard-interactive` to allow both pubkey and the interactive 2FA prompt. + - `PasswordAuthentication no` to disable password-based authentication. + - `ChallengeResponseAuthentication yes` to enable 2FA interactive challenge. + - `UsePAM yes` to use the 2FA authenticator libpam module. +7. Restart the SSH daemon with `sudo systemctl restart sshd.service`. + +Note: SSH root login will be disabled implicitly by requiring pubkey authentication and having no pubkeys listed in `/root/.ssh/authorized_keys`. + +https://www.digitalocean.com/community/tutorials/how-to-set-up-multi-factor-authentication-for-ssh-on-ubuntu-16-04 + +### Disabling 2FA +Some use cases (such as programmatic access) demand 2FA be disabled. +Some day we'll figure out how to allow specific keys to bypass the 2FA requirement. But until then, + +1. Edit the file `/etc/pam.d/sshd` and comment out the line `auth sufficient pam_google_authenticator.so nullok` +2. Edit the file `/etc/ssh/sshd_config` and find the `AuthenticationMethods` configuration. Replace the value `publickey,keyboard-interactive` with `publickey`. + +### SSH Key Management +The process for managing SSH keys should work as follows: + +1. SSH access to hosts should be controlled via keys listed in `~/.ssh/authorized_keys`. +2. One key should map to one user on one device. +3. When authorizing a key, review existing authorized keys and remove as appropriate. +4. Device keys should be stored under the "SSH Keys" folder in the password manager. The pubkey should be the "password" for easy copying, and the private component should be added as an attachment. + +## Patching and Updating +In the interest of proactively mitigating security risks, we try to keep packages up to date. We have two main concerns for patching: host packages, and docker images. Each of these have their own concerns and are handled separately. + +### Host Packages via Unattended Upgrades +Since Debiant 9, the `unattended-upgrades` and `apt-listchanges` are installed by default. + +1. Install the packages with `sudo apt-get install unattended-upgrades apt-listchanges`. +2. Create the default automatic upgrade config with `sudo dpkg-reconfigure -plow unattended-upgrades` + +By default, we will get automatic upgrades for the distro version default and security channels (e.g. `bullseye` and `bullseye-security`) with the `Debian` and `Debian-Security` labels. + +https://wiki.debian.org/UnattendedUpgrades + +### Debian Version Upgrade +When the time comes for a major version upgrade on a Debian system, we take the following steps as soon as realistic. + +1. Update the current system with `sudo apt-get update && sudo apt-get upgrade && sudo apt-get full-upgrade`. +2. Switch the update channel for APT sources. + 2a. Export the name of the new version codename to a variable with `NEW_VERSION_CODENAME=bookworm` (bookworm as an example). + 2b. `for file in /etc/apt/sources.list /etc/apt/sources.list.d/*; do sudo sed "s/$VERSION_CODENAME/$NEW_VERSION_CODENAME/g" $file; done`. +3. Clean out old packages and pull the new lists `sudo apt-get clean && sudo apt-get update` +4. Update to most recent versions of all packages for new channel with `sudo apt-get upgrade && sudo apt-get full-upgrade` +5. Clean out unnecessary packages with `sudo apt-get autoremove`. +6. Reboot the host to finalize changes with `sudo shutdown -r now`. + +Note: If migrating from Debian versions <12 to versions >=12, add the following repos (in addition to `main`) after step 2a: `contrib non-free non-free-firmware`. + +https://wiki.debian.org/DebianUpgrade + +### Docker Images +As of now, we have no automated process or tooling for updating Docker images. + +We usually update Docker images one stack at a time. For example, we'll update `calibre-web` on `fighter`: + +1. Navigate to the directory of the stack. `cd ~/homelab/fighter/config/calibre-web` +2. Check the images and tags to be pulled with `docker-compose config | grep image` +3. Pull the latest version of the image tagged in the compose file `docker-compose pull` +4. Restart the containers to use the new images with `docker-compose up -d --force-recreate` + +Note: We can update one image from a stack by specifying the name of the service. E.g. `docker-compose pull forwardauth` \ No newline at end of file