CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED ·
When the conversation can’t tell which booking the customer means, RiskGate asks instead of guessing.
Transcript0:53 · narrated
Offline replay of public τ²-bench logs (Sierra, MIT licence) · simulated customers · not a live deployment
01 · What happened
The situation
τ²-bench airline, task 42, a 2026 run of Gemini 3 Flash. The customer may be double-booked and isn't sure which same-day flights are extra. The agent guesses: cancel both May 22 flights. She says yes.
02 · The miss
Why the obvious safeguard didn’t catch it
It cancelled the Boston flight she needed. Her yes didn't help, because she didn't know either; across the 2026 airline runs, a yes preceded 199 of 209 wrong writes.
03 · RiskGate
List, play forward, decide
Before the cancel runs, RiskGate lists every booking she could mean: her six active reservations.
The cancel is right in one of them and wrong in five.
Not safe in all of them, so it is not certified at the 10% limit. The agent asks which flight instead of guessing.
04 · At scale
What held, and the price
Across 8 frontier agents, wrong writes stayed under the 10% limit. The price, stated plainly: it asked on 35–80% of writes, depending on the agent.
199 / 209wrong writes after a yes
1 of 6bookings where the cancel is right
35–80%of writes asked (the price)
Name one action your agent isn’t allowed to take today.
Send us its traces and what happened afterwards. We’ll show how often it could have run safely, and what it would have stopped.