CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED ·
A customer’s “yes” doesn’t check the database write. RiskGate checks every reading of the conversation before the write runs.
Transcript0:54 · narrated
Offline replay of public τ²-bench logs (Sierra, MIT licence) · real agents, simulated customers · not a live deployment
01 · What happened
The situation
τ²-bench retail, task 74, a 2026 leaderboard run of Claude Sonnet 4.5. The customer asks for her refund on her credit card. The agent tells her it must go back to PayPal, the original payment method. Unsure, she says yes, and the write sends $218.84 to PayPal.
02 · The miss
Why the obvious safeguard didn’t catch it
The rule the agent quoted was wrong. Confirming with the customer feels like a safeguard, but she can only confirm what she is told; her yes checked the agent's sentence, not the write.
03 · RiskGate
List, play forward, decide
Before the write, RiskGate asks what else she could mean: 15 readings of the conversation are still plausible.
It plays the write (refund to paypal_5914760) forward in each. In the reading where she meant her Visa card, the write is wrong.
Wrong in a plausible reading, so it is not certified: the agent asks one question before any money moves.
04 · At scale
What held, and the price
Across 8 frontier agents on the 2026 leaderboard, 355 of 363 wrong writes came right after a yes. Calibrated per agent at a 10% limit, RiskGate kept all 12 agents (2025 and 2026) under it. The price is that it asks more often.
355 / 363wrong writes followed a yes
12 / 12agents under the 10% limit
$218.84sent to the wrong method
Name one action your agent isn’t allowed to take today.
Send us its traces and what happened afterwards. We’ll show how often it could have run safely, and what it would have stopped.