Independent project · personal lab

Homelab AI

Ansible configuration-as-code for a small self-hosted AI inference lab: two Intel Arc GPU workers on one host behind an authenticated gateway, with Vault-managed secrets, scheduled reconciliation, validation evidence, and rollback paths.

Reviewed at commiteb9c25eon 2026-10-25. Repository:github.com/mikeholownych/homelab-ai.

Problem and intended users

Running local inference on workstation-class hardware creates a long list of state that is easy to lose: GPU drivers, container runtimes, model versions, secrets, firewall rules, and the gap between what is running and what was intended. Homelab AI treats Git as the desired state and uses Ansible to converge a personal lab toward it.

The intended user is one operator managing a small number of self-hosted hosts. The inventory defines two nodes, but only the Dell Precision 5820 (ai-5820-01) is described as commissioned at the v1.0.0 release; the Lenovo ThinkStation P620 (ai-p620-01) remains an uncommissioned future profile. This is not presented as a multi-tenant platform, managed service, or customer deployment.

What the implementation does

The repository holds inventory, hardware profiles, version pins, and policies alongside Ansible roles and playbooks. Convergence covers base OS boundaries, users and SSH access, GPU and container runtime setup, model serving, patching, upgrades, validation, and evidence collection.

On the qualified host, two independent containerized llama.cpp SYCL workers run one per GPU. An orchestrator_gateway reverse proxy provides bearer-token authentication, circuit breaking, request queueing, semantic caching, and W3C distributed tracing. The README records that vLLM was retired on this platform after empirical qualification favored llama.cpp SYCL; that is a platform-specific result, not a general claim about either engine.

HashiCorp Vault is the secret authority. Playbooks obtain short-lived AppRole tokens, and no plaintext secrets or Ansible Vault files are committed. Scheduled reconciliation runs from a deployed snapshot through systemd timers with flock single-instance protection. Validation probes are read-only and produce JSON artifacts; evidence bundles include a manifest, host and PCI topology, validation and benchmark results, and SHA256SUMS.

Recovery is documented rather than implied. Component upgrades track current, candidate, and previous_known_good versions, with a rollback mode and a runbook for reverting a commit and re-converging.

Engineering practices evidenced in the repository

  • Configuration-as-code with secrets excluded from Git and a single secret authority.
  • Hash-pinned Python dependencies, version-pinned Ansible collections, and hash-pinned observability artifacts fetched with checksum verification.
  • Independent validation separated from convergence, with schema-validated JSON evidence and checksums.
  • Drift detection and scheduled reconciliation, with fail-closed classification when validation is missing or NOT_TESTED.
  • Least-privilege service account, explicit SSH key proof before disabling password authentication, and opt-in firewall policy with explicit management CIDRs.
  • GPU telemetry sampled per device and sensor, with warning and critical thresholds set below the benchmark abort guardrail.
  • Rollback runbooks and version tracking for high-risk components.

The quality workflow runs make quality, which covers linting, tests, syntax checks, and idempotency checks, using pinned harness images for Ubuntu 24.04 and 26.04. GitHub recorded that workflow as successful at the reviewed commit on 2026-10-25 (about 16 minutes). That is a historical result for one commit and the scope defined in the workflow; it does not prove a live inference service or a production outcome.

Maturity, validation scope, and limitations

The latest published release is v1.0.0 (2026-10-14). One host is qualified. The second node is uncommissioned, and clustering is described as future configuration rather than an enabled capability.

  • The lab is single-host in practice. There is no claim of high availability, multi-tenant isolation, enterprise deployment, or customer workload.
  • The continuous observability stack is provided as a role but installs fail-closed and is gated on a commissioning checklist. Operational failure alerts and a GPU textfile metrics bridge are implemented; the full Grafana/VictoriaMetrics/Loki/Tempo stack is not claimed as deployed.
  • Vault is the secret authority, but service keys are rendered into root-readable runtime files at convergence. Revocation therefore takes effect at the next convergence or service restart; the README describes moving fully to refreshed systemd credentials as a planned refinement.
  • The repository’s Phase 14 readiness report is dated 2026-09-29 and names an earlier HEAD. I treat it as historical context, not current validation.
  • The repository’s findings summary is dated 2026-06-14 and records unresolved items from that review. I did not independently re-verify every item for this page.
  • I did not execute the repository or run its test suite. Validation statements here are limited to the repository’s own recorded results and documentation.
  • I found no evidence of customer deployment, commercial adoption, third-party certification, or an external security audit. I do not claim any of those.

My contribution

I created and maintain the repository. At the reviewed commit, all 348 commits in the history are attributed to my author addresses. I have not attempted to attribute individual roles beyond that commit history.

Evidence reviewed

These links are pinned to the reviewed commit so the descriptions above can be checked against the same source.

This is an independent engineering project. It is not a client engagement and does not describe an employer’s systems or outcomes.