When Your AI Server Goes Rogue: Building a GPU Watchdog for Self-Hosted LLMs

There's a particular sound that tells you something is wrong with your home server: the fans. When my mini PC's cooling system kicked into high gear last weekend, I knew something had changed. The problem? I hadn't touched it in hours, and neither had anyone else in my household.

Self-hosting an LLM with Ollama on my AMD Ryzen AI MAX+ 395 has been rewarding. I get privacy, no subscription fees, and a model that works completely offline. The tradeoff is that when something breaks, there's no platform operations team pinging me with incident reports. I'm both the developer and the on-call engineer.

The Diagnosis

My first instinct was paranoia—had someone compromised the machine and turned it into a crypto miner? A quick check with ollama ps revealed the truth: a model was stuck in "Stopping..." state, unable to unload and free the GPU. The process was hung, but the hardware thought it had work to do.

I confirmed this by reading the GPU utilization directly from the kernel at /sys/class/drm/card1/device/gpu_busy_percent. The numbers were damning: 89% GPU utilization, pulling a steady 85 watts—with zero inference requests happening. This had been running for approximately 20 hours. The standard ollama stop command had no effect, though sudo systemctl restart ollama cleared it immediately.

This is a known bug in Ollama where the GPU stays pegged at full utilization even when the model has no actual work to perform. It's not dangerous, but it's wasteful—and on a machine running 24/7, those watts add up on your electricity bill.

Building the Watchdog

A manual diagnosis works when you're paying attention, but I needed automation. I didn't want to be the first line of defense against unnecessary power consumption. The solution was a lightweight watchdog that monitors for this specific failure condition.

The logic needed to be narrow to avoid false positives. Here's the approach I settled on:

A cron job runs every 5 minutes and checks two conditions: whether a model process is loaded and what the GPU utilization reads. An unhealthy state only triggers when both conditions are true simultaneously—a model loaded AND GPU at or above 70% utilization. High GPU usage during active inference is completely normal, so this specifically targets high usage during idle periods.

The watchdog requires the unhealthy state to persist for 15 minutes before alerting. This eliminates false positives from legitimate long-running requests. Any healthy reading resets the counter, ensuring the alert only fires on genuinely stuck processes rather than transient spikes.

When the alert fires, it sends a push notification via ntfy.sh directly to my phone with the diagnosis and the one-line fix command. After sending, the watchdog suppresses further alerts for one hour—no stream of duplicate notifications while you're trying to fix the problem.

Two Design Decisions Worth Noting

First, the watchdog alerts but never restarts automatically. A person can distinguish a stuck runner from a legitimate long job in seconds, and I didn't want automation killing active work. The alert tells me exactly what's wrong and gives me the command to fix it.

Second, if the alert fails to send, the watchdog doesn't mark the event as handled. The next run will retry instead of going silent. This matters—silent failures in monitoring are worse than no monitoring at all.

The Setup

The cron job lives in /etc/cron.d/ollama-watchdog as a system drop-in, which means it survives user logouts and has the right permissions to read system metrics. The script writes two timestamp files that persist across runs, tracking whether we're in a healthy state or accumulating unhealthy readings.

The beauty of this approach is its simplicity. No external monitoring service, no subscription, no complex configuration. Just a cron job, a couple of files, and a push notification when something goes wrong.

Results

Since implementing this watchdog, a stuck Ollama runner gets discovered, evaluated, and reset within 15 minutes of occurrence. Before this, a hung process could run silently for 20+ hours. Now I get a phone notification with clear guidance, whether I'm sitting at the machine or halfway across the city.

If you're running self-hosted AI infrastructure—whether it's Ollama on a mini PC or vibe coding on Vibe Hosting—consider what failure modes you're not watching for. The best infrastructure is the kind that tells you when something breaks before you notice it yourself.

Self-hosting gives you control. But control only works if you're watching.