CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED ·
One alert often fits several root causes. Act only if the fix works for every cause that still fits; otherwise check first or ask a person.
Transcript0:54 · narrated
Real telemetry (RCAEval) and real agent runs (ITBench) · fix outcomes rule-derived · offline analysis
01 · What happened
The situation
A real fault-injection run from RCAEval on Online Boutique: the shop's front end slows down, and four more services jump at the same moment. A scripted agent picks the biggest jump and wants to add more copies.
02 · The miss
Why the obvious safeguard didn’t catch it
The loudest metric isn't the cause. Real SRE agents struggle here too: on ITBench, a ReAct SRE agent (GPT-OSS-120B) named the right root cause 15 times in 85.
03 · RiskGate
List, play forward, decide
RiskGate keeps every cause that still fits the telemetry; here, four of 55 candidates.
More copies would help in only one of them.
Not safe for every cause, so it asks a person. The real fault was disk stress on the product catalog; more copies wouldn't have fixed it.
04 · At scale
What held, and the price
Over 99 held-out runs, the list kept the real cause 96 times, usually with 2 causes (median). The fix outcomes are rule-derived, so this is evidence about the diagnosis step, not a live remediation result.
96 / 99held-out runs kept the real cause
15 / 85SRE agent diagnoses right (ITBench)
2causes on a typical list
Name one action your agent isn’t allowed to take today.
Send us its traces and what happened afterwards. We’ll show how often it could have run safely, and what it would have stopped.