Selfhost Stacks

Production-ready self-hosted infrastructure running on Proxmox, managed with Infrastructure as Code (Terraform) and GitOps workflows. Currently hosting 90+ containerized services across media streaming, home automation, development tools, and AI workloads.

Memory headroom is the binding constraint. The host has ~29GB usable and typically runs at 24–25GB. New stacks should set explicit mem_limit values rather than running unbounded. (Corrected 2026-08-16: this README previously claimed 64GB. Actual is 32GB physical, ~29GB usable after the Radeon 890M iGPU carve-out.)

🏗️ Architecture

This repository is split into two clean domains:

• infra/ — Terraform IaC for Proxmox host configuration and LXC/VM provisioning
• stacks/ — Docker Compose stacks deployed to LXC containers

Platform

Resource	IP	Purpose	Specs
Proxmox Host	172.16.1.158	Type-1 hypervisor	AMD Ryzen AI 9 HX PRO 370 (24 cores), 32GB RAM (~29GB usable), ZFS pool
LXC 100 (selfhost)	172.16.1.159	Primary Docker runtime	Unprivileged LXC, GPU passthrough (AMD Radeon 890M)
VM 102 (Cerebro)	172.16.1.160	AI/ML workloads	Ubuntu 24.04, GPU passthrough, Ollama, OpenClaw

Storage

• ZFS Pool: /mnt/fast - High-performance NVMe storage (host)
• Mount into LXC/VM: /mnt/fast/stacks (repo), /mnt/fast/appdata (persistent data)
• Service Account: apps:apps (UID/GID 568:568) for container ownership

Quota'd datasets

Dataset	Mountpoint	Quota	Why
fast/transcode	/mnt/fast/transcode	50 G	Jellyfin's transcode cache. Bound into the container at /cache/transcodes
tank/media/Recordings	/mnt/tank/media/Recordings	250 GB	Dispatcharr server DVR; Jellyfin's TV Recordings library. No automatic deletion

fast/transcode exists because an anonymous Docker volume holding Jellyfin's transcode cache grew to 19 GB on LXC 100's root filesystem and drove / to zero mid-Phase-3. Anonymous volumes live under /var/lib/docker/volumes — i.e. on / — and nothing in the compose file named them, so nothing bounded them.

The dataset is declared in infra/jellyfin-transcode-dataset.tf and tuned for disposable sequential data: recordsize=1M, compression=off (segments are already-compressed H.264/HEVC), sync=disabled, atime=off.

It uses quota, not refquota, deliberately: quota counts descendants and returns ENOSPC, which is the errno every log grep in this estate is built on. refquota returns EDQUOT and would be invisible to them.

Standing check: scripts/check-jellyfin-transcode.sh asserts the quota, the mount shape, and five Jellyfin retention settings, and exits non-zero on drift. It is folded into scripts/quick-health-check.sh.

Per-app datasets

/mnt/fast/appdata itself is still not a mountpoint: it is a plain directory on LXC 100's ext4 root, and only its children are ZFS datasets. The stacks below moved off that root on 2026-10-22 (quick task 261022-cg6). Each has its own dataset, its own LXC bind mount and a full send to the backup pool, so the nightly incremental carries it.

Dataset	LXC mount	recordsize	Quota	Why
fast/appdata/teslamate	mp32	16K	—	TeslaMate Postgres (the car's whole history), Grafana, import dir. Was three named volumes on /
fast/appdata/trek	mp33	128K	—	TREK SQLite (data/) and uploads. Was two named volumes on /
fast/appdata/monitoring	mp34	128K	25G	Prometheus TSDB, Grafana, node-exporter textfile dir. The one continuously growing tenant, so it is bounded
fast/appdata/airtrail	mp35	16K	—	AirTrail Postgres and uploads
fast/appdata/silo	mp36	16K	—	Silo (dormant: silo-db and silo stay stopped)
fast/appdata/bookorbit	mp37	128K	—	BookOrbit covers and book-bucket (data/). Created on its own dataset 2026-10-22 (261022-wp7)
fast/appdata/bookorbit/postgres	mp38	16K	—	BookOrbit Postgres (18/docker)

n8n's Postgres moved to /mnt/fast/appdata/automation/n8n-postgres, inside the existing fast/appdata/automation dataset, with no new mount. It had been on /automation/n8n-postgres, because ${APPDATA_DIR} was undefined in automation/.env.

16K follows fast/appdata/immich/postgres, the estate's other Postgres-tuned dataset. There is no quota on the Postgres datasets on purpose: ENOSPC under Postgres WAL is a PANIC.

A new stack needs all four of these, in this order, or its state lands on /:

1. zfs create fast/appdata/<name> on atlantis (owner 568:568, mode 0755).
2. pct set 100 -mpN /mnt/fast/appdata/<name>,mp=/mnt/fast/appdata/<name>. A new mpN hotplugs live with no LXC reboot (measured 2026-10-22). Next free indices: mp29, mp30, mp39+.
3. Append the matching mount_point block to infra/lxc-selfhost.tf in host index order. Never terraform apply it (see the lifecycle note there).
4. Do a full zfs send | zfs recv to backup/appdata/<name>. backup-incremental.sh skips a dataset that has no target on the backup pool, so it is not backed up until this is done.

Media and TV — high-level design

flowchart LR
    A[HDHomeRun aerial tuner] --> D[Dispatcharr]
    P[IPTV provider] --> D
    D -->|Live TV| T[TiviMate]
    D -->|Scheduled server recording| R[Recordings storage: 250 GB]
    R --> J[Jellyfin: TV Recordings]
    D -->|Alternative live TV client| J

Loading

TiviMate is the preferred live-TV interface. Schedule recordings in Dispatcharr's TV Guide or DVR page; the server records even when the television is off. Watch saved programmes in Jellyfin's TV Recordings library. Recording storage has a 250,000,000,000-byte quota (about 233 GiB); reaching it stops new writes rather than deleting older recordings. The IPTV account permits one upstream channel, so recording and watching different IPTV channels simultaneously is constrained. The aerial tuner is independent.

TiviMate's Record button still uses its own recorder. This server DVR setup does not provide live pause/rewind in TiviMate; that integration remains outstanding. Implementation, storage mappings and verification are in the media LLD and Dispatcharr runbook. The recording dataset is managed by infra/dispatcharr-recordings.tf.

📁 Repository Structure

selfhost-stacks/
├── infra/                      # Terraform infrastructure as code
│   ├── host.tf                # Proxmox host configuration
│   ├── lxc-selfhost.tf        # LXC 100 provisioning (Docker runtime)
│   ├── lxc-cerebro.tf         # VM 102 provisioning (AI workloads)
│   ├── providers.tf           # Terraform provider configuration
│   └── variables.tf           # Input variables and defaults
├── stacks/
│   ├── selfhosted/            # Services for LXC 100
│   │   ├── traefik/          # Reverse proxy + SSL (core dependency)
│   │   ├── arrs/             # Media automation (Sonarr, Radarr, Prowlarr, etc.)
│   │   ├── media/            # Streaming (Jellyfin, Jellyseerr)
│   │   ├── automation/       # Home automation (n8n, Oxidized, PWPush)
│   │   ├── openwebui/        # AI chat interface
│   │   ├── immich/           # Photo management
│   │   ├── code-server/      # Web-based VS Code
│   │   └── ...               # 15+ additional stacks
│   └── cerebro/              # Services for VM 102
│       └── borg/             # OpenClaw AI gateway
├── scripts/                   # Operational utilities
│   ├── pg-migrate.sh         # PostgreSQL major-version migration (dump/restore)
│   ├── setup-teleport.sh     # Teleport agent bootstrap
│   └── check-renovate.sh     # Renovate dependency validation
└── .github/
    └── workflows/
        └── compose-validate.yaml  # CI: Validate compose files on PRs

🚀 Quick Start

Prerequisites

• Proxmox VE 8.x+ installed on host
• Terraform 1.10+ on your local machine
• SSH access to Proxmox host (root@172.16.1.158)

1️⃣ Infrastructure Deployment

Deploy Proxmox host configuration and provision LXC containers:

cd infra
terraform init
terraform plan
terraform apply

What this does:

• Configures ZFS storage pool on Proxmox host
• Creates unprivileged LXC 100 (selfhost) with Docker, GPU passthrough, and mounts
• Provisions VM 102 (Cerebro) with Ubuntu 24.04, GPU passthrough, Ollama, and OpenClaw
• Sets up service account (apps:apps) and permissions
• Maps AMD Radeon 890M GPU devices (video:44, render:110)

2️⃣ Stack Deployment

SSH into the LXC and deploy stacks from the repository:

# On your local machine - sync repository to server
ssh root@172.16.1.159
cd /mnt/fast/stacks
git pull origin main

# Deploy foundational stack (Traefik reverse proxy)
docker compose -f stacks/selfhosted/traefik/compose.yaml up -d

# Deploy application stacks
docker compose -f stacks/selfhosted/arrs/compose.yaml up -d
docker compose -f stacks/selfhosted/media/compose.yaml up -d
docker compose -f stacks/selfhosted/automation/compose.yaml up -d
docker compose -f stacks/selfhosted/openwebui/compose.yaml up -d

Compose files reference:

• Each stack is modular and can be deployed independently
• Stacks share the t3_proxy network managed by Traefik
• Environment variables stored in .env files (use .env.sample templates)

🔄 Automated Updates with Renovate

This repository uses Renovate for automated dependency updates:

• ✅ Auto-merge patch updates (1.2.3 → 1.2.4) after 1-hour stability period
• ✅ Auto-merge minor updates (1.2.0 → 1.3.0) after 3-hour stability period
• ⚠️ Manual review for major updates (1.x → 2.x) to prevent breaking changes
• 🔒 Security updates merge immediately regardless of stability period

Configuration: See renovate.json

Dashboard: Dependency Dashboard Issue #3 - View all pending updates

Container Version Pinning Strategy

• Pinned versions (e.g., jellyfin:10.11.6) are tracked by Renovate and auto-updated
• :latest tags are ignored by Renovate (manual updates required)
• Branch tags (:develop, :stable) are ignored by Renovate

Best Practice: Pin to specific semver tags to enable automatic updates while maintaining stability.

🧪 CI/CD

GitHub Actions automatically validate compose files on every pull request:

• ✅ Syntax validation with docker compose config
• ✅ Image availability checks
• ✅ Environment variable reference validation
• ✅ Prevents broken configurations from merging

Workflow: .github/workflows/compose-validate.yaml

🎯 Common Workflows

Deploy a New Stack

1. Create compose file in stacks/selfhosted/<stack-name>/compose.yaml
2. Pin container versions (avoid :latest)
3. Create .env.sample template for required variables
4. Document service in compose file comments
5. Commit and push to GitHub
6. SSH to LXC and deploy: docker compose -f stacks/selfhosted/<stack-name>/compose.yaml up -d

PostgreSQL Major Version Migration

PostgreSQL major version upgrades (e.g. PG17 → PG18) require a dump/restore cycle — data is not forward-compatible. All postgres volumes now mount at /var/lib/postgresql (PG18 layout); data lives in a versioned subdir (18/docker/).

ssh root@172.16.1.159
cd /mnt/fast/stacks

# Pull updated compose files (with new image tags)
git pull

# Run the migration script for a single instance
bash scripts/pg-migrate.sh n8n

# Or migrate all instances in correct order
bash scripts/pg-migrate.sh all

The script handles dump, old-data removal, PG18 init, restore, and service restart. Backups are retained in /tmp/pg-backups/ on the host.

Note: pgvector/pgvector images are blocked from Renovate major-version auto-merge — they must be migrated manually using the same procedure.

Update Existing Stack

ssh root@172.16.1.159
cd /mnt/fast/stacks
git pull origin main
docker compose -f stacks/selfhosted/<stack-name>/compose.yaml pull
docker compose -f stacks/selfhosted/<stack-name>/compose.yaml up -d

View Running Containers

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

Check Stack Logs

docker compose -f stacks/selfhosted/<stack-name>/compose.yaml logs -f

Rebuild Infrastructure

cd infra
terraform plan
terraform apply

🔐 Security & Access

Traefik Reverse Proxy

• SSL/TLS: Automatic certificate management with Let's Encrypt
• Authentication: Authelia integration for protected services
• Middleware: Rate limiting, IP whitelisting, basic auth available

Service Access Patterns

• Public: No authentication (e.g., Jellyfin for family, PWPush for sharing)
• Protected: Authelia SSO (admin dashboards, management interfaces)
• Internal Only: No Traefik route, LXC network only (databases, caches)

Environment Variables

Sensitive data stored in .env files (gitignored):

• Database passwords
• API keys
• Secret keys
• JWT tokens

Template files: .env.sample committed to repository as documentation

🎮 GPU Hardware Acceleration

AMD Radeon 890M GPU is passed through to LXC 100:

Device Mapping:

• /dev/dri/renderD128 (group render:110)
• /dev/dri/card1 (group video:44)

Configuration: See infra/lxc-selfhost.tf for LXC device mapping

Services:

• Immich: hardware-accelerated photo/video processing — active
• Jellyfin: ⚠️ hardware transcoding is DISABLED, deliberately. Jellyfin transcodes in software (libx264). /dev/dri is still mapped into the container, but Jellyfin does not use it.

⚠️ Do not "re-enable" Jellyfin hardware acceleration

Jellyfin's live config is HardwareAccelerationType: none and EnableHardwareEncoding: false. This is the D-30 amdgpu mitigation, applied after the 2026-08-31 outage that cost ~6 hours.

The failure mode is a kernel oops in the amdgpu HMM path triggered by GPU userptr mappings. Its blast radius is counter-intuitive: dockerd reads /proc/*/smaps, so Traefik and every *.deercrest.info service die while direct-macvlan containers keep serving. It presents as /proc/pressure/io full ≈ 96% with an idle CPU and kernel tainted bit 7 — not as load average, which lags and false-greens after reboot. Proxmox kernel 7.0.14-8 does not carry a fix.

High CPU during a Jellyfin transcode is therefore correct behaviour, not a fault to investigate. scripts/check-jellyfin-transcode.sh reports both values on every run so drift is visible; scripts/disable-jellyfin-hwaccel.sh is the recovery lever.

🩺 Health Checks

scripts/quick-health-check.sh is the estate's single entry point. Run it from the workstation; it ssh's to LXC 100 and folds in four subordinate checks:

bash scripts/quick-health-check.sh            # exits 0 healthy, 1 on any violation
REMOTE_TIMEOUT=300 bash scripts/quick-health-check.sh   # slower link / busy host

Check	Asserts
check-music-freeze.sh	Jellyfin's Music library cannot write .nfo/.jpg/.lrc into the library
check-music-consumers.sh	the NFS export and its consumers; Jellyfin addressing facts
check-jellyfin-transcode.sh	the 50 G transcode quota, the mount shape, and five Jellyfin retention values
check-drift.sh	which running containers are on an image tag git no longer pins; the unresolvable-pin set; unhealthy and created containers

Design rules these checks follow

They exist because a previous generation of checks printed a green tick while blind. Three rules came out of that, and new checks should honour them:

1. Fail closed, and keep "could not look" distinct from "nothing is wrong." A check that cannot reach the host reports UNKNOWN and exits non-zero — never 0 problems found.
2. Bound every remote command Linux-side. The workstation is macOS and has no GNU timeout, so bounds go inside the remote command string (REMOTE_TIMEOUT, default 120s). A killed command surfaces as its own condition via exit 124. ⚠️ timeout N cmd | wc -l does not work — timeout signals only the first stage, wc counts the empty stream and the pipeline exits 0. The remote string needs set -o pipefail and the ssh return code must be read. ${PIPESTATUS[0]} does not rescue it: an assignment is a simple command, not a pipeline.
3. Assert, don't report. A value that is merely printed cannot fail the run. Anything that must not drift gets an explicit expectation and increments FAILURES when it moves.

Anything asserted must also be proven able to fail — drive it red once, deliberately, rather than trusting that it would. Every EXPECT_* in check-jellyfin-transcode.sh is env-overridable precisely so a negative control needs no file edit.

A fourth rule, which check-drift.sh is the first check here to need

4. If a check runs unattended, its own staleness is part of what it asserts. The other three only ever fire when somebody types them, so "did it run?" is answered by the fact that you are reading the output. check-drift.sh also runs on an hourly systemd timer, writing to a node-exporter textfile — and a timer that dies produces silence that looks exactly like health, with the last measurement frozen in place and every downstream rule quietly reading a snapshot. So it emits selfhost_image_drift_last_success_timestamp_seconds, writes it only on a successful run, and leaves the previous file byte-identical on a failure so it goes stale rather than fresh-and-wrong. A dedicated Grafana rule alerts on that staleness, and all three rules carry noDataState: Alerting so an absent metric alerts instead of going quiet.

   Note rule 3 is deliberately not applied to the drift count itself: v1 is alert-only, so asserting it would make this entry point permanently red for a condition nothing in the repo can clear. See IMAGE_DRIFT_PROMOTED in quick-health-check.sh.

📊 Current Stacks

Media & Streaming

• Jellyfin - Media server (movies, TV, music)
• Jellyseerr - Content request management
• Jellystat - Analytics and statistics
• Wizarr - User invitation system

Media Automation (Arrs)

• Sonarr - TV show management
• Radarr - Movie management
• Lidarr - Music management
• Bazarr - Subtitle management
• Prowlarr - Indexer manager
• Autobrr - IRC announce grabber
• Bookshelf (container readarr) - Book and audiobook management; Readarr fork on the hardcover (Hardcover.app) flavour since 2026-10-22; metadata from the self-hosted rreading-glasses (internal only, http://rreading-glasses:8788) since 2026-10-23

Books

• BookOrbit - Ebook and audiobook reader at books.deercrest.info, own login, library read-only (replaced Booklore 2026-10-22; see stacks/selfhosted/bookorbit/README.md)
• Audiobookshelf - Audiobooks and podcasts at audiobooks.deercrest.info

Development

• Code Server - Web-based VS Code
• OpenWebUI - AI chat interface (Ollama frontend)

Automation

• n8n - Workflow automation
• Oxidized - Network device backup
• Teleport - Access control proxy

Photo Management

• Immich - Google Photos alternative

Infrastructure

• Traefik - Reverse proxy and SSL termination
• Authelia - SSO authentication
• Docker Socket Proxy - Secure Docker API access

See full inventory: 70+ containers across 20+ stacks

📚 Documentation

• CLAUDE.md - AI Agent guidance for repository operations
• DEPLOY.md - Detailed deployment procedures
• STANDARDS.md - Coding and configuration standards

🔧 Troubleshooting

Container fails to start

docker compose -f stacks/selfhosted/<stack>/compose.yaml logs

Permission issues

Ensure files are owned by apps:apps (568:568):

chown -R 568:568 /mnt/fast/appdata/<stack>

GPU not accessible

Verify device mapping in LXC:

ls -l /dev/dri/
groups apps  # Should include 'video' and 'render'

Traefik routing issues

Check Traefik dashboard: https://traefik.deercrest.info

Renovate not creating PRs

Check Dependency Dashboard for ignored or blocked updates

🤝 Contributing

1. Pin container versions (avoid :latest)
2. Include .env.sample for new stacks
3. Document service purpose in compose file comments
4. Test compose validation locally: docker compose config
5. Create descriptive commit messages

📜 License

Private repository - All rights reserved

---

Maintained by: Damian Flynn
Last Updated: June 2026
Repository: github.com/DamianFlynn/selfhost-stacks