CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED ·
The booking is clear; which cancellation rule applies to it is not. RiskGate reads the reservation first and lets the refund run only if the policy allows it in every reading that still fits.
Transcript0:55 · narrated
Offline replay of public τ²-bench logs (Sierra, MIT licence) · simulated customers · verdicts: riskgate 1.0.0rc2 · not a live deployment
01 · What happened
The situation
τ²-bench airline, task 45, a 2026 run of Claude Sonnet 4.5. A family emergency: the customer wants an immediate cancellation and a full refund. The agent cites a “silver member” rule the policy doesn't have. She says yes and cancel_reservation runs.
02 · The miss
Why the obvious safeguard didn’t catch it
The benchmark's expected behaviour was no write at all. The reservation ID was never in doubt, so a which-booking check passes; the doubt is which clause of the policy applies, and the API doesn't check cancellation rules.
03 · RiskGate
List, play forward, decide
RiskGate asks which clause could apply: booked in the last 24 hours, cancelled by the airline, business cabin, or insurance with a covered reason. The refund is safe in four readings, not all.
So it reads the record first (get_reservation_details, read-only): basic economy, no insurance, booked ten days ago, nothing cancelled.
No clause fits: REJECT, nothing runs (1.9 ms). The contrast, a real correct cancel from task 14: same fare, booked 23 hours earlier. Clause one holds: CERTIFY, it runs.
04 · At scale
What held, and the price
Across 8 agents, 8.9–26.5% of airline writes were wrong, mostly right after a yes. On held-out tasks behind RiskGate, per-task wrong writes fell from 12.7–36.3% to 4.6–5.9%, under the 10% limit. The price: it asks on 35–80% of writes.
12.7–36.3% → 4.6–5.9%wrong writes, per agent
1.9 msread, then reject
35–80%of writes asked
Name one action your agent isn’t allowed to take today.
Send us its traces and what happened afterwards. We’ll show how often it could have run safely, and what it would have stopped.