CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED · CERTIFY · REJECT · PROBE · ASK · CALIBRATED CONSEQUENCE CHECKS FOR THE ACTIONS OF AI AGENTS · NO LLM ON THE CERTIFY PATH · FAIL CLOSED ·
“I don’t recognise this payment” is four different situations, and blocking the card is wrong in three of them. Blocking is also an action.
Transcript0:55 · narrated
Banking77 (PolyAI, CC BY 4.0) quotes · simulated accounts and ledger · real riskgate verdicts · not affiliated with any bank
01 · What happened
The situation
1,658 of 13,083 real online-banking messages in the public Banking77 dataset are about a charge the customer didn't expect. On one real message, the dispute bot's draft is to block the card and raise a dispute. It never looked the payment up.
02 · The miss
Why the obvious safeguard didn’t catch it
A block feels like the safe default, but a hotel pre-authorisation hold, a subscription under an unfamiliar name or an add-on card purchase are not fraud. Blocking those cuts the customer off and files a false dispute.
03 · RiskGate
List, play forward, decide
RiskGate asks what this could be: an authorisation hold, a subscription, an add-on card, or fraud. Block is safe in one world; explain-and-watch in three.
Neither is safe everywhere, so it looks first: a read-only transaction lookup.
A hotel hold? Block skipped; explain certified, the card stays active. Fraud signals? Block and dispute certified, at once. Every decision leaves a receipt.
04 · At scale
What held, and the price
On simulated held-out cases, blocking every time was wrong 70.7% of the time. Behind RiskGate, 72.5% of cases ran with no human and 0.73% were certified wrong, under a 5% budget.
70.7% → 0.73%wrong actions (simulated)
72.5%no human needed
1,658real “unexpected charge” messages
Name one action your agent isn’t allowed to take today.
Send us its traces and what happened afterwards. We’ll show how often it could have run safely, and what it would have stopped.