From the incident log

Notes

Every problem on the homelab gets an entry: symptom, cause, fix, rule. Most are three lines. These three earned a few paragraphs, because the interesting part was how I was wrong before I was right.

2026-08-29

Six weeks of inference on the CPU

Local models on the homelab were slow enough that I'd stopped using them. I had an explanation ready: the GPU has 2 GB of memory, the models want more, so of course it's slow. That explanation was comfortable, it was plausible, and it was wrong for six weeks.

What actually happened is that the Compose service for the inference container never declared runtime: nvidia. The Docker daemon had the NVIDIA runtime registered, and I'd verified that, so I assumed containers could use it. They can't. Registering the runtime on the daemon is necessary but not sufficient; each service has to ask for it. The container had never seen the GPU at all.

symptom: Local models painfully slow. Assumed the 2 GB GPU was the limit.
cause: No runtime: nvidia on the service. The container ran on CPU for six weeks.
fix: Add the runtime plus NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES. Prompt evaluation went from 77 tokens/s to 730–870.
rule: Before blaming hardware, prove the container can see it: docker exec <container> nvidia-smi. Ten seconds.

The lesson isn't about NVIDIA. It's that a plausible explanation stopped me from running a ten-second check. The thing I'd "verified" was one layer above the thing that mattered. Now, when something is slow, the first question is whether the component can see the resource I think it's using, and the answer has to come from inside the component.

2026-07-28

A clean exit is not the same as working

The game-streaming host died after every reboot. Not crashed: systemctl status said active, running, and also said Tasks: 0. The port it should have been listening on was never bound. I spent the first pass on the video encoder, because that's what streaming services usually break on, and got nowhere.

The real cause was ordering. The service starts before the desktop's screen-capture portal is ready, finds nothing to capture, and exits. It exits with code zero. That single fact defeated every recovery mechanism I had, because Restart=on-failure only fires on failure, and nothing had failed as far as systemd could tell.

symptom: Service shows running after boot; port never binds; zero tasks.
cause: Capture portal not ready at start. The process exits clean, so the restart policy never triggers. Not the encoder.
fix: Order the unit after the portal and audio stack, add a start delay, and add a systemd timer that tests whether the port is listening and restarts the unit if not.
rule: Check outcomes, not process state. A health check has to test the thing you actually need, which is a bound port, not a PID.

I ended up with three fixes and only the third one mattered. The ordering and the delay reduce how often it happens; the timer is what makes it not matter when it does. That's the version I'd reach for first next time: make the system self-correcting against the symptom, then reduce the cause if it's cheap.

2026-08-30

Measure before building

I built an AI agent on the homelab to run a weekly coursework report and post it to Notion. It took about six hours. It lasted twenty-four.

The teardown was triggered by one bug. The script that created Notion rows printed "Successfully created row" for rows that did not exist. I lost hours chasing a phantom authentication error before reading the script itself: its success check was if [ $? -eq 0 ] on a curl call. That tests whether curl ran, not whether the API accepted anything.

symptom: Tool reports success. Rows don't exist. Hours chasing a 401 that wasn't the problem.
cause: Success was inferred from the exit code of the HTTP client, not from the response body.
rule: Any script that writes to an API checks the response, not the exit code. A tool that lies about success is worse than no tool.
rule: Verify independently when something reports success.

The bug was fixable. The larger problem wasn't. The agent needed constant hand-holding, and the actual job, a scheduled report, doesn't need a model at all; the script already produced readable output, and the agent was only formatting it. I'd built the interesting thing instead of the necessary thing, on an assumption about what would help, without measuring first.

The next day I installed a different agentic tool, confirmed it worked, and removed it before spending a single credit. There was no workload yet: no coursework in Notion, no syllabi, no reporter. An agent with nothing to act on is a hobby. It goes back on when there's something for it to do, and that decision now has a trigger written next to it instead of a feeling.