Jafner/Jafner.net
1
0
Fork 0
Code Issues 16 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Feature: Init stack.nix for monitoring.

Browse Source 
- Nuke everything except uptime-kuma.
This commit is contained in:
Joey Hafner 2025-02-01 01:08:22 -08:00
parent fc2264331b
commit 66d1928b81
Signed by: Jafner
GPG Key ID: 6D9A24EF2F389E55
16 changed files with 16 additions and 11395 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
homelab/stacks/monitoring/.env
View File

@ -1 +0,0 @@
DOCKER_DATA=/home/admin/data/monitoring

145
homelab/stacks/monitoring/README.md
View File

@ -1,145 +0,0 @@
# Grafana
## Updating Configuration File
The Grafana config is edited by providing overrides in `$DOCKER_DATA/custom.ini`, which maps to `/etc/grafana/grafana.ini` inside the container.
The `custom.ini` file stores secrets in plain text, so we can't keep it in version control. But I've included snippets for reference below:
### Basic Server Config
```ini
[server]
domain = grafana.jafner.net
root_url = %(protocol)s://%(domain)s/
force_migration = true
```
### Configure Auth to Sign In via Keycloak
```ini
[auth]
oauth_auto_login = true
[auth.anonymous]
enabled = true
[auth.generic_oauth]
name = OAuth
icon = signin
enabled = true
client_id = grafana.jafner.net
client_secret = **************************
scopes = email openid profile
empty_scopes = false
auth_url = https://keycloak.jafner.net/realms/Jafner.net/protocol/openid-connect/auth
token_url = https://keycloak.jafner.net/realms/Jafner.net/protocol/openid-connect/token
api_url = https://keycloak.jafner.net/realms/Jafner.net/protocol/
signout_redirect_url = https://grafana.jafner.net
```
### Configure Email Sending via SMTP (Protonmail)
```ini
[smtp]
enabled = true
host = smtp.protonmail.ch:587
user = noreply@jafner.net
password = ****************
from_address = noreply@jafner.net
from_name = Grafana
startTLS_policy = OpportunisticStartTLS
```
# Monitoring Specification
Monitors are split into three types: Host, Application, and IoT
All monitors use a Prometheus exporter.
## Hosts
| Name | IP (if static) | OS | Exporter |
|:----:|:--------------:|:--:|:--------:|
| Router | 192.168.1.1 | Linux 4.14) | [node_exporter](https://github.com/prometheus/node_exporter) |
| Server | 192.168.1.23 | Linux 5.10) | [node_exporter](https://github.com/prometheus/node_exporter) |
| Seedbox | 192.168.1.21 | Linux 5.10) | [node_exporter](https://github.com/prometheus/node_exporter) |
| NAS | 192.168.1.10 | FreeBSD 12.2) | ???
| PiHole | 192.168.1.22 | Linux 5.10) | [node_exporter](https://github.com/prometheus/node_exporter) |
## Applications
| Name | Address(es) | Exporter |
|:----:|:-------:|:--------:|
| Minecraft | e6.jafner.net, vanilla.jafner.net | [mc-monitor](https://github.com/itzg/mc-monitor)
| GitLab | gitlab.jafner.net | [GitLab Integrated Exporter](https://docs.gitlab.com/ee/administration/monitoring/prometheus/gitlab_metrics.html)
| Traefik | traefik.jafner.net | [Prometheus - Traefik.io](https://doc.traefik.io/traefik/observability/metrics/prometheus/) |
| Deluge | jafner.seedbox:52000, jafner.seedbox:52100, jafner.seedbox:52200 | [deluge_exporter](https://github.com/tobbez/deluge_exporter) |
| Plex | plex.jafner.net | [Tautulli](https://github.com/Tautulli/Tautulli) and [tautulli-exporter](https://github.com/nwalke/tautulli-exporter), or [plex_exporter](https://github.com/arnarg/plex_exporter) |
| PeerTube | peertube.jafner.net | [Add a Prometheus Exporter - GitHub Issue](https://github.com/Chocobozzz/PeerTube/issues/3742) |
| WordPress | nvgm.jafner.net | [wordpress-exporter](https://github.com/aorfanos/wordpress-exporter) |
| SabNZBD | sabnzbd.jafner.net | [sabnzbd_exporter](https://github.com/msroest/sabnzbd_exporter) |
| Uptime Kuma | uptime.jafner.tools | [Prometheus Integration - Uptime Kuma Wiki](https://github.com/louislam/uptime-kuma/wiki/Prometheus-Integration) |
| PiHole | jafner.pi1 | [pihole-exporter](https://github.com/eko/pihole-exporter) |
| ZFS | nas.jafner.net | [zfs_exporter](https://github.com/pdf/zfs_exporter) |
## IoT
| Name | Hostname | Assigned IP | Note |
|:----:|:--------:|:-----------:|:----:|
| tasmota-1 | tasmota-F6441E-1054 | 192.168.1.50 |
| tasmota-2 | tasmota-F6D7D3-6099 | 192.168.1.51 |
| tasmota-3 | tasmota-F6F062-4194 | 192.168.1.52 |
# Adding Loki and Promtail
Followed [this guide from Techno Tim](https://docs.technotim.live/posts/grafana-loki/).
Non-tracked changes include:
1. `docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions` to install the Loki docker plugin.
## Instrumenting: Daemon-Level Logging
Edit `/etc/docker/daemon.json` to add the following block:
```json
{
"log-driver": "loki",
"log-opts": {
"loki-url": "http://localhost:3100/loki/api/v1/push",
"loki-batch-size": "400",
"loki-retries": "1",
"loki-timeout": "2s"
}
}
```
NOTE: All logging will fail if the Loki container is inaccessible. This may cause the Docker daemon to lock up. These parameters are applied when a container is created, so all containers must be destroyed to resolve the issue.
NOTE: The batch size here is in lines for *all docker logs*.
## Instrumenting: Per-Container Logging
Add the following logging parameter to each main-service container within a stack.
```yml
services:
<some-service>:
logging:
driver: loki
options:
loki-url: http://localhost:3100/loki/api/v1/push
loki-batch-size: "50"
loki-retries: "1"
loki-timeout: "2s"
keep-file: "true"
```
NOTE: The batch size here is in lines for *only the selected container*.
See [loki log-opts](https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/#supported-log-opt-options) for list of available configuration options for loki logging driver.
See [docker-compose logging](https://docs.docker.com/compose/compose-file/compose-file-v3/#logging) for Docker-compose logging reference.
## Instrumenting: Default Docker Logging
Per: [Docker docs](https://docs.docker.com/config/containers/logging/configure/)
> The default logging driver is `json-file`.
The configuration options for the `json-file` logging driver are [here](https://docs.docker.com/config/containers/logging/json-file/).
Docker-compose adds a few labels to containers it starts. This feature is not comprehensively documented, but here: [Compose Specification](https://docs.docker.com/compose/compose-file/). And we can see what labels are added by default by simply looking at a deployed application (wg-easy):
| Label Key | Value |
|:---------:|:-----:|
| `com.docker.compose.config-hash` | `f75588baa1056ddc618b1741805d2600b4380e13c5114106de6c8322f79dfd3f` |
| `com.docker.compose.container-number` | `1` |
| `com.docker.compose.oneoff` | `False` |
| `com.docker.compose.project` | `wireguard` |
| `com.docker.compose.project.config_files` | `docker-compose.yml` |
| `com.docker.compose.project.working_dir` | `/home/joey/homelab/jafner-net/config/wireguard` |
| `com.docker.compose.service` | `wg-easy` |
| `com.docker.compose.version` | `1.29.2` |
These are *labels* on the container, which are distinct from *tags* in the actual json log payload. Log tags are [documented here](https://docs.docker.com/config/containers/logging/log_tags/).

75
homelab/stacks/monitoring/docker-compose.yml
View File

@ -1,81 +1,10 @@
services:
grafana:
image: grafana/grafana-oss:latest
container_name: monitoring_grafana
restart: "no"
networks:
- monitoring
- web
user: "0"
volumes:
- ${DOCKER_DATA}/grafana:/var/lib/grafana
- ${DOCKER_DATA}/custom.ini:/etc/grafana/grafana.ini
labels:
- traefik.http.routers.grafana.rule=Host(`grafana.jafner.net`)
- traefik.http.routers.grafana.tls.certresolver=lets-encrypt
- traefik.http.routers.grafana.tls.options=tls12@file
- traefik.http.routers.grafana.middlewares=securityheaders@file
prometheus:
image: prom/prometheus:latest
container_name: monitoring_prometheus
user: 1001:1001
networks:
- monitoring
- web
ports:
- 9090:9090
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ${DOCKER_DATA}/prometheus:/prometheus
restart: "no"
command:
- "--config.file=/etc/prometheus/prometheus.yml"
labels:
- traefik.http.routers.prometheus-monitoring.rule=Host(`prometheus.jafner.net`)
- traefik.http.routers.prometheus-monitoring.tls.certresolver=lets-encrypt
- traefik.http.routers.prometheus-monitoring.middlewares=traefik-forward-auth-privileged@file
exporter-ping:
image: czerwonk/ping_exporter:latest
container_name: monitoring_exporter-ping
restart: "no"
networks:
- monitoring
volumes:
- ./exporter-ping/config.yml:/config/config.yml
exporter-docker:
image: prometheusnet/docker_exporter:latest
container_name: monitoring_exporter-docker
restart: "no"
networks:
- monitoring
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
labels:
- traefik.enable=false
exporter-pihole:
image: ekofr/pihole-exporter:latest
container_name: monitoring_exporter-pihole
env_file:
- path: ./exporter-pihole.env
required: true
- path: ./exporter-pihole_secrets.env
required: false
restart: "no"
networks:
- monitoring
labels:
- traefik.enable=false
uptime-kuma:
image: louislam/uptime-kuma:latest
container_name: monitoring_uptime-kuma
restart: "no"
volumes:
- $DOCKER_DATA/uptime-kuma:/app/data
- $APPDATA/uptime-kuma:/app/data
- /var/run/docker.sock:/var/run/docker.sock
networks:
- web
@ -84,7 +13,5 @@ services:
- traefik.http.routers.uptime-kuma.tls.certresolver=lets-encrypt
networks:
monitoring:
external: true
web:
external: true

2
homelab/stacks/monitoring/exporter-pihole.env
View File

@ -1,2 +0,0 @@
PIHOLE_HOSTNAME=192.168.1.32
PORT=9617

29
homelab/stacks/monitoring/exporter-ping/config.yml
View File

@ -1,29 +0,0 @@
targets:
- 8.8.8.8 # google primary
- 1.1.1.1 # cloudflare primary
- 9.9.9.9 # quad9 primary
- 192.168.1.1 # local gateway
- jafner.net # own public ip
- tukw-dsl-gw76.tukw.qwest.net # Owned by CENTURYLINK-US-LEGACY-QWEST (ASN: 209) (https://www.findip-address.com/63.231.10.76)
- tukw-agw1.inet.qwest.net # Owned by CENTURYLINK-US-LEGACY-QWEST (ASN: 209) (https://www.findip-address.com/63.226.198.89)
- sea-edge-15.inet.qwest.net # Owned by CENTURYLINK-US-LEGACY-QWEST (ASN: 209) (https://www.findip-address.com/205.171.0.59)
- 4.68.38.173 # Owned by LEVEL3 (ASN: 3356) (https://www.findip-address.com/4.68.38.173)
- 4.69.219.210 # Owned by LEVEL3 (ASN: 3356) (https://www.findip-address.com/4.69.219.210)
- 4.30.140.62 # Owned by LEVEL3 (ASN: 3356) (https://www.findip-address.com/4.30.140.62)
- 172.71.144.3 # Owned by CLOUDFLARENET (ASN: 13335) (https://www.findip-address.com/172.71.144.3)
- 192.168.1.10 # Primary NAS
- 192.168.1.11 # Backup NAS
- 192.168.1.32 # Local DNS server
dns:
refresh: 1m
nameserver: 1.1.1.1
ping:
interval: 5s
timeout: 3s
history-size: 5
payload-size: 32
options:
disableIPv6: true

2942
homelab/stacks/monitoring/grafana-dashboards/Home.json
View File

File diff suppressed because it is too large Load Diff

5457
homelab/stacks/monitoring/grafana-dashboards/Hosts.json
View File

File diff suppressed because it is too large Load Diff

1767
homelab/stacks/monitoring/grafana-dashboards/Services.json
View File

File diff suppressed because it is too large Load Diff

818
homelab/stacks/monitoring/grafana-dashboards/Smart Plugs.json
View File

@ -1,818 +0,0 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 7,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "kwatth"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 0
},
"id": 5,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[30d])",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Total Usage (last month)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "kwatth"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 7,
"y": 0
},
"id": 6,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[7d])",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Total Usage (last week)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "kwatth"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 14,
"y": 0
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[24h])",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Total Usage (last 24h)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "currencyUSD"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 5
},
"id": 11,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[30d])*0.08192",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Power Cost (last month)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "currencyUSD"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 7,
"y": 5
},
"id": 12,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[7d])*0.08192",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Power Cost (last week)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "currencyUSD"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 14,
"y": 5
},
"id": 13,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "auto"
},
"pluginVersion": "8.5.2",
"targets": [
{
"exemplar": true,
"expr": "delta(tasmota_energy_power_kilowatts_total[24h])*0.08192",
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Power Cost (last 24h)",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"axisSoftMin": 0,
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "watt"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "tasmota_energy_power_active_watts{instance=\"tasmota-1:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Joey's PC"
}
]
},
{
"matcher": {
"id": "byName",
"options": "tasmota_energy_power_active_watts{instance=\"tasmota-2:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "YA Fridge"
}
]
},
{
"matcher": {
"id": "byName",
"options": "tasmota_energy_power_active_watts{instance=\"tasmota-3:80\", job=\"tasmota\"}"
},
"properties": [
{
"id": "displayName",
"value": "Freezer"
}
]
}
]
},
"gridPos": {
"h": 11,
"w": 17,
"x": 0,
"y": 10
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"exemplar": true,
"expr": "tasmota_energy_power_active_watts{}",
"instant": false,
"interval": "",
"legendFormat": "",
"queryType": "randomWalk",
"refId": "A"
}
],
"title": "Power Draw",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "7z5wqRXnz"
},
"gridPos": {
"h": 11,
"w": 3,
"x": 17,
"y": 10
},
"id": 8,
"options": {
"content": "| Service | Charge |\n|:-------:|:------:|\n| Energy | $0.045351/kWh |\n| Distribution | $0.036569/kWh |\n| Total | $0.08192/kWh |",
"mode": "markdown"
},
"pluginVersion": "8.5.2",
"targets": [
{
"channel": "plugin/testdata/random-20Hz-stream",
"exemplar": true,
"expr": "",
"filter": {
"fields": [
"Time",
"Value"
]
},
"interval": "",
"legendFormat": "",
"queryType": "measurements",
"refId": "A"
}
],
"title": "Power Rates",
"type": "text"
}
],
"refresh": "5s",
"schemaVersion": 36,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-6h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Smart Plugs",
"uid": "A5-cSus7z",
"version": 14,
"weekStart": ""
}

34
homelab/stacks/monitoring/loki-config.yml
View File

@ -1,34 +0,0 @@
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /loki-logs
storage:
filesystem:
chunks_directory: /loki-logs/chunks
rules_directory: /loki-logs/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
storage_config:
filesystem:
directory: /loki-logs
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093

75
homelab/stacks/monitoring/prometheus.yml
View File

@ -1,75 +0,0 @@
global:
scrape_interval: 60s
scrape_configs:
# scrape hosts
- job_name: 'fighter'
scrape_interval: 5s
static_configs:
- targets: ['192.168.1.23:9100']
- job_name: 'druid'
scrape_interval: 5s
static_configs:
- targets: ['143.110.151.123:9100']
- job_name: 'wizard'
scrape_interval: 5s
static_configs:
- targets: ['192.168.1.1:9273']
# scrape applications
- job_name: 'prometheus-self'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'traefik'
scrape_interval: 5s
static_configs:
- targets: ['traefik:8080']
metrics_path: "/metrics"
- job_name: 'exporter-pihole'
scrape_interval: 5s
static_configs:
- targets: ['exporter-pihole:9617']
metrics_path: "/metrics"
- job_name: 'exporter-minecraft'
scrape_interval: 5s
static_configs:
- targets: ['exporter-minecraft:8080']
- job_name: 'exporter-plex'
scrape_interval: 5s
static_configs:
- targets: ['exporter-plex:9594']
- job_name: 'exporter-sabnzbd'
scrape_interval: 5s
static_configs:
- targets: ['exporter-sabnzbd:9387']
- job_name: 'exporter-ping'
scrape_interval: 5s
static_configs:
- targets: ['exporter-ping:9427', 'jafner.tools:50418']
- job_name: 'exporter-docker'
scrape_interval: 5s
static_configs:
- targets: ['exporter-docker:9417','jafner.tools:50417']
- job_name: 'exporter-qbittorrent'
scrape_interval: 60s
scrape_timeout: 30s
static_configs:
- targets: ['exporter-qbittorrent:8000']
# scrape IoT devices
- job_name: 'tasmota'
scrape_interval: 5s
static_configs:
- targets: ['192.168.1.50','192.168.1.51','192.168.1.52']

10
homelab/stacks/monitoring/promtail-config.yml
View File

@ -1,10 +0,0 @@
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push

19
homelab/stacks/monitoring/scripts/diskstatus.sh
View File

@ -1,19 +0,0 @@
#!/usr/bin/env sh
SMARTCTL=/usr/local/sbin/smartctl
DISKS=$(/sbin/sysctl -n kern.disks | cut -d= -f2)
for DISK in ${DISKS}
do
TEMP=$(${SMARTCTL} -l scttemp /dev/${DISK} | grep '^Current Temperature:' | awk '{print $3}')
HEALTH=$(${SMARTCTL} -H /dev/${DISK} | grep 'test result:' | cut -d: -f2 | sed 's/^[ \t]*//')
if [ -z != ${TEMP} ] && [ -z != ${HEALTH} ]
then
JSON=$(echo ${JSON}{\"disk\":\"${DISK}\",\"health\":\"${HEALTH}\",\"temperature\":${TEMP}},)
fi
done
JSON=$(echo ${JSON} | sed 's/,$//')
echo [${JSON}] >&1

3
homelab/stacks/monitoring/scripts/forgepc.sh
View File

@ -1,3 +0,0 @@
#!/bin/bash
docker exec e6 rcon-cli forge entity list "minecraft:player" >&1

19
homelab/stacks/monitoring/scripts/forgetps-to-json.sh
View File

@ -1,19 +0,0 @@
#!/bin/bash
# this script converts the output of the "forge tps" command (in the form of the .forgetps file) into json for sending to influxdb
# by default it reads from stdin and outputs to a .forgetps.json file
while IFS= read -r line; do
if [ "$line" != "" ]; then
DIM=$(echo -n "$line" | awk '{print $2}')
if [ "$DIM" = "Mean" ]; then
DIM="Overall"
fi
TPT=$(echo "$line" | grep -oE 'Mean tick time: .+ms' | awk '{print $4}')
TPS=$(echo "$line" | grep -oE 'Mean TPS: .+' | awk '{print $3}')
JSON+=\{$(echo \"dim\":\"$DIM\",\"tpt\":$TPT,\"tps\":$TPS)\},
fi
#done < .forgetps # inputs from .forgetps file
done <$1 # inputs from file passed via stdin
JSON=$(echo ${JSON} | sed 's/,$//')
#echo [${JSON}] >&1 # outputs to stdout
echo [${JSON}] > .forgetps.json # uncomment this to output to file

15
homelab/stacks/monitoring/stack.nix Normal file
View File

@ -0,0 +1,15 @@
{ sys, ... }: let stack = "monitoring"; in {
home-manager.users."${sys.username}".home.file = {
"${stack}" = {
enable = true;
recursive = true;
source = ./.;
target = "stacks/${stack}/";
};
"${stack}/.env" = {
enable = true;
text = ''APPDATA=${sys.dataDirs.appdata}/${stack}'';
target = "stacks/${stack}/.env";
};
};
}