An agent that has to prove every claim it makes

The premise of Agent K was a complaint: an agent that investigates an outage can hallucinate a root cause just as confidently as it can find one. Ours had to work in a way where that failure mode is structurally impossible, not merely discouraged by a system prompt.

That turned into three rules we enforced in software.

No claim without evidence. A root-cause claim carries a list of SigNoz links — a trace, a log, a metric, a deployment. The report renderer drops any claim whose evidence list is empty. The model can hallucinate all it likes; nothing unsupported survives to the report.

No action without budget. Before any action runs, a policy function checks SLO breach, burn rate, allowlist membership, cooldown, confidence threshold, whether the cause is deployment related, and sandbox scope. The allowlist has exactly one entry: roll back to the previous version. The model recommends; the code decides.

No self without telemetry. Every LLM call, token count, cost, MCP query, hypothesis and policy verdict becomes a span. A loop breaker stops repeated queries and a cost watchdog kills over-budget investigations. The agent is observable in the same dashboard as the app it is watching.

Underneath sits the thing being monitored: a FastAPI RAG support service backed by Postgres 16 with pgvector, local MiniLM embeddings, and an OpenAI-compatible client pointed at whichever free tier still had quota that day.

The part I would defend hardest is the four seeded incidents. Two are deployment-caused, and the policy gate permits a rollback. Two are not — an injected retrieval delay and a database pool exhaustion — and the correct behaviour is to refuse and escalate to a human. A demo where the agent always acts proves nothing. Half of ours is the agent declining to.

The lesson worth carrying out of the week: "the model decided not to" is not a safety property. Only code that runs before the action is.