This article describes a new benchmark that measures the reliability of AI agentic automation for AML alert triage: the investigation a bank makes of every alert its transaction monitoring raises.

• The benchmark describes a fictional bank, its customers and their transactions, its written investigation procedure and disposition rules, and 82 alerts with their expected results.

• Thunk.AI built an execution and evaluation harness, so that any agentic implementation can be measured on the same alerts, mock systems and checks.

• Thunk.AI implemented the benchmark on the Thunk.AI platform and ran it on the four models. For comparison, the same models ran the whole procedure as a single AI step, with the same documents and mock systems, without the platform.

1: The goals of the benchmark

AI agents can take on much of the manual work in business processes. In a regulated process, though, an incorrect decision has real consequences, and enterprise customers adopt AI automation only when they trust it to follow the process accurately and consistently. They describe this as a need for AI reliability.

Most AI benchmarks measure a model on a task. Thunk.AI's "HiFi" series measures AI agents doing enterprise business processes, so that customers can compare agentic platforms and decide whether, and how, to automate a process with them. Earlier benchmarks in the series measured document workflows (September 2025), IT service management (February 2026) and drug safety case intake (October 2026).

This benchmark has four goals:

1. Represent a regulated, high-stakes process that every bank runs every day: the investigation and disposition of transaction-monitoring alerts.

2. Publish the process, the data and the checks, so that anyone can run the benchmark on another implementation and compare.

3. Measure what matters to a bank's financial crimes team: whether each disposition is right, whether an error could leave suspicious activity unreported, and whether the case record is accurate.

4. Measure how far a platform can take AI reliability with non-frontier models, chosen for cost, sovereignty or the freedom to run them anywhere, against stronger models and against the same models without the platform.

The latest frontier models handle a well-specified workflow like this one with little help. But there are good reasons not to use them: cost, data sovereignty, and a growing range of smaller and open-weight models that an enterprise can run where it chooses.

2: The AML alert triage domain

Banks must monitor their customers' transactions for money laundering, terrorist financing and sanctions evasion. A transaction-monitoring system applies rules to every account (cash deposits just below the $10,000 reporting threshold, money sent on as soon as it arrives, wires to high-risk jurisdictions, activity far above the customer's peers) and raises an alert when a rule fires.

Most alerts are false positives, but each must be investigated. An investigator reviews the alert, gathers the customer's record and history, decides whether the activity is suspicious, and records a disposition. When it is suspicious, the bank must file a Suspicious Activity Report (SAR) within a fixed time. A missed SAR is a compliance failure that can lead to enforcement action and fines, and law enforcement never learns of the activity. An unnecessary SAR, or unnecessary monitoring, wastes investigators' and reviewers' time, and in volume it buries the reports that matter.

The investigation has three parts:

1. Intake. Check the alert's data: which listed transactions exist and are on the customer's own accounts, and which are data issues.

2. Context and reasoning. Gather the customer's KYC record, account history, peer comparison and prior cases; decide which red flags the activity raises, and whether a bank record explains them.

3. Disposition. Clear the alert, monitor the customer, or escalate and draft the SAR with its filing deadline, and send the case to the right reviewer.

The work is detailed and unforgiving. Thresholds are exact ($8,000 to $9,999.99, three deposits within five days, at least 80% sent on within three days), time windows are exact (a SAR in the past 24 months, a false-positive case closed in the past 12), and an explanation counts only when a bank record covers the same counterparty and a consistent amount. The bank's records are written as a compliance officer would write them, in prose ("Invests in Northwind Alpha Fund Ltd (Cayman Islands); subscriptions of $50,000–$100,000 twice a year"), so the investigator must read them carefully. A reliable system applies the written rules to every alert, every time.

3: Benchmark methodology

3.1. Workflow

The benchmark starts from a sample standard operating procedure (SOP) with seven steps: review the alert; assemble the context; reason about the activity; recommend the disposition; draft the SAR; have an investigator review the case; and record the reviewer's feedback. The SOP and its reference documents (the bank's disposition rules, its monitoring rules, its intake and review procedures, and its SAR drafting guidelines) are written in plain English. An implementation receives these documents and nothing else.

3.2. Metrics

Every alert has one right answer under the written rules. Each alert is checked on 23 facts (1,886 checks in all), for example "The disposition is Amber" or "The red flags raised are C and E". A fact the result does not state fails its check. Each alert then falls into exactly one of five categories:

The headline metric is the Weighted Score. Each alert starts at 100 points and loses points for what went wrong: 50 for a safety error; 25 for an unnecessary SAR, and 15 for unnecessary monitoring or a workflow that did not finish; and 2 for each wrong detail, up to 25. The Weighted Score is the 82 alerts' total as a percentage of the best possible total.

3.3. Data set

A fictional bank, Larchmont Harbor Bank, has 86 customers (individuals and businesses) with KYC records, 356 transactions over four months, peer benchmarks and prior cases. The 82 alerts are 25 standard cases, 21 edge cases that sit exactly on a rule's threshold, and 36 hard cases with a trap an investigator must reason through (an unlisted deposit that completes structuring, an explanation that appears only in the customer's own memo, a bank record about a different counterparty, a listed transaction on another customer's account). 27 alerts need a SAR, 28 need monitoring and 27 are false positives.

3.4. Evaluation harness

Mock systems stand in for the bank's customer records, transaction history, peer benchmarks, case management, SAR document store and investigator review. Each alert runs as an independent workflow instance. Every check is an exact test of a recorded value (a color, a date, a list of red flags, a name); a scorer assigns each alert its category and points.

4: Benchmark implementation results

4.1. Results on four AI models

Each configuration processed all 82 alerts once. "With Thunk.AI" is the final workflow design (section 4.4); gpt-5.1 and gemini-3.5-flash ran the design one iteration earlier, which differs only in the checks added for the weaker models (see the notes on method).

Costs are at each provider's standard list price (gpt-5.1 ran on the discounted flex tier and is repriced to standard).

¹ mistral-medium-3's times are inflated by throttling at the model provider, which limited how many requests it would serve at once; they do not reflect the model's speed.

4.2. Detailed breakdown

With Thunk.AI, the four models made two disposition errors in 328 alert runs, both unnecessary escalations: gpt-5.1 placed one false positive under monitoring, and mistral-medium-3 drafted one unnecessary SAR. The detail errors were a mis-recorded transaction list on gemini-3.5-flash and five alerts on mistral-medium-3, in its intake, SAR and review fields.

Field-level pass rates, by part of the investigation (checks passed of those run):

The pattern is consistent: without the platform, models gather the facts well (intake and context) and then lose the thread when they must apply many exact rules to them at once (reasoning, SAR and review).

4.3. Comparison: without Thunk.AI

Each model received the whole SOP and all of its documents in a single AI step, with the same mock systems as tools, and recorded its result as one structured object. Its only help was a list of the facts it must record, so that it was graded on exactly the same checks. The kinds of mistake it made:

• Over-escalation. Every model escalated alerts the rules clear or only monitor: 9 to 40 unnecessary escalations per model. gpt-5-mini drafted 39 unnecessary SARs.

• Missed SARs. gpt-5.1, gemini-3.5-flash and mistral-medium-3 missed SARs or cleared alerts that needed monitoring (4, 5 and 7 alerts), mostly by crediting a bank record that does not cover the activity.

• Incomplete or unreadable results. gpt-5-mini finished 17 alerts without a usable result (11 empty, 6 not valid JSON). The single step has nothing that stops an agent from finishing with its work undone.

4.4. What the platform adds: from the SOP to a reliable thunk

The benchmark began with the SOP turned into a thunk as directly as possible: the seven steps in order, the SOP's own step text, the reference documents searchable, and the bank's systems as tools. Every later change kept the seven steps, their order and the rules, and added structure that the Thunk.AI application model provides. None of it is specific to an alert. The table lists the constraints the final design has that the initial one did not.

The starting point, measured. The initial thunk (the SOP's seven steps as written, with the reference documents and the bank's systems, and none of the constraints above) ran on gpt-5-mini over the 82 alerts. It finished 39; the other 43 stopped to ask a person something the rules already answer, such as whether an Amber alert needs a SAR or which reviewer to send a case to. It recorded the right disposition on 31 alerts, made 20 safety errors and 28 unnecessary escalations, and recorded no disposition on 3. The final thunk, on the same model, finished all 82 and got every disposition right. The seven steps alone did not make gpt-5-mini reliable; the constraints did.

How each constraint showed up in the results. On gpt-5.1, the full 82 alerts scored 94.1 when the facts were first typed and the rules moved into code, 97.2 once each step had its own fields (the disposition step could no longer override the rules), and 99.7 once outputs were required and explanations were judged by code. The weaker models needed the last three constraints. On a 10-alert sample of harder-than-average alerts, gpt-5-mini scored 78.4 with required outputs alone (half its workflows stopped because a step skipped a lookup and left a required field empty), 93.4 once the step instructions named every lookup, and 100.0 on all 82 alerts once the rules tool also checked its records. mistral-medium-3 went from 89.7 to 99.4 the same way.

Why these constraints work. They follow the principles of Thunk.AI's reliability architecture, described in AI Reliability Principles:

• Decomposition. The process runs as small steps rather than one large task, each with one job, its own fields and its own tools. A model asked to do one well-defined thing at a time keeps to the rules far more consistently than one asked to carry a long procedure in its head.

• Maximizing constraints. Typed fields, fixed values and required outputs narrow the space of answers, so there are fewer ways to be wrong, and a wrong or missing answer is caught at the step that made it, not at the end.

• Separating deterministic logic from AI decisions. Wherever the rules state a test, the test runs as code; the AI does what only AI can (query the systems, read the records, write the investigation and the SAR), and code checks what it recorded. This is what lets a smaller model do the work: it no longer has to apply a dozen exact rules correctly at once.

4.5. Summary of key findings

1. Non-frontier models delivered excellent reliability on Thunk.AI: gpt-5-mini scored 100.0 and mistral-medium-3 99.4, with no safety errors, level with the stronger models (99.7 and 100.0).

2. Without the platform, the same non-frontier models scored 68.8 and 78.2. With the SOP's steps but none of the platform's constraints, gpt-5-mini finished fewer than half of the alerts.

3. The platform closed the gap between models, from 22.6 points to 0.6.

4. Reliability did not require an expensive model: gpt-5-mini cost about $0.03 per alert on the workflow, roughly a tenth of gpt-5.1 at list price.

5. The gain comes from structure the platform supplies (section 4.4), not from prompting: every constraint restates the written procedure, and none is specific to an alert.

Notes on method.

• Each configuration processed the 82 alerts once.

• The design with Thunk.AI was refined over several iteration cycles. Each cycle looked at the failures and applied more of the platform's mechanisms (section 4.4). None captured any knowledge of the alerts or their expected results.

• The weaker models, gpt-5-mini and mistral-medium-3, needed one extra iteration that added two constraints: the bank's records copied verbatim and checked by the rules tool, and step instructions that name every lookup and the fields it fills, with the platform telling a step which required outputs are still missing. gpt-5.1 and gemini-3.5-flash already scored 99.7 and 100.0 without them.

• Without Thunk.AI, the single step records one free-form result. A result that was not valid JSON as a whole was read up to its first malformed field, and two spellings were accepted for the typology and the reviewer role.

5: Appendix: Benchmark details

The complete benchmark is published for full transparency: the SOP and its reference documents, the fictional bank and its data, all 82 alerts with their expected results, the checks, and how they are scored. They are at the Benchmark definition site.

5.1. Workflow design

The SOP's seven steps, in order:

1. Alert intake and review: review the alert's type, date, customer, transactions and triggering rule, and flag missing or inconsistent data.

2. Assemble context: the customer profile, historical behavior, peer group comparison and prior alerts.

3. Conduct investigation reasoning: the typology, the red flags and any documented explanation.

4. Recommend disposition: clear (Green), monitor (Amber) or escalate (Red).

5. Draft SAR: for an escalated alert, draft the Suspicious Activity Report and record its filing deadline.

6. Human review and approval: send the case to an investigator for review.

7. Capture feedback: record the reviewer's decision.

The disposition rules settle what the SOP leaves open: which listed transactions count; the seven red flags (A structuring, B a sanctioned jurisdiction or party, C a high-risk or secrecy jurisdiction, D pass-through, E activity inconsistent with the customer's profile, F a prior SAR in 24 months, G a PEP without a documented source of funds) and their exact thresholds; the jurisdiction lists; when a documented explanation covers a red flag; the disposition (Red if A or B is raised, or two or more red flags are unexplained; Amber if exactly one is; Green otherwise); the typology; the SAR deadline (30 days from the alert); and the reviewer (the BSA Officer for a Red alert, the AML Team Lead otherwise).

5.2. Scope

The disposition rules are the fictional bank's internal rules, written to test whether an investigator applies written rules; they are not regulatory determinations. Real financial crimes teams also weigh what this benchmark leaves out: customer due diligence refresh, negative news and open-source research, network analysis across related customers, requests for information from the customer, continuing-activity SARs, and the investigator's own judgment of suspicion beyond any written rule.

5.3. The alerts

Each of the 82 alerts tests one rule or trap. The full list, with each alert's expected disposition and SAR deadline, is in the Benchmark definition site and on the results page.

5.4. Ground truth and evaluation

The facts of each alert (the customer, the transactions, the peer benchmark, the prior cases) were generated together with the mock data. Every label (the red flags raised and explained, the disposition, the typology, the SAR deadline, the reviewer and the data issues) was then derived from those facts by a published script that follows the written rules, so it matches the rules exactly. A second, separate implementation of the rules reproduces the same dispositions.

The expected results were then reviewed against the practice of AML investigation. AI models, acting as independent AML reviewers, reviewed every expected result and every rule; Thunk.AI decided each finding. No human AML professional has reviewed the expected results. The review changed the rules (Syria, whose sanctions program was revoked in 2025, was taken off the sanctions list; pass-through and profile inconsistency were defined exactly; the kinds of record that can explain a red flag, and when an amount is consistent, were stated), dated every peer benchmark before its alert, named the reviewers, changed the expected disposition of one alert, and added 7 alerts. Every result in this article is graded against the reviewed ground truth.

5.5. Benchmark variations

The benchmark can be run with other AI models, to see how far a platform narrows the differences between them. It can also be varied to model a specific bank: other monitoring rules and thresholds, other jurisdiction lists, other alert mixes and volumes, or other review policies.

5.6. Acceptable implementation guidelines

A valid implementation of this benchmark should follow these guidelines:

• The SOP, the reference documents and the alerts must not be augmented with alert-specific detail by a human implementer. They may be reformatted to suit the platform.

• The workflow may be refined, modularized and given tools, provided it keeps the SOP's steps, order and rules. Tools may apply the written rules; they must not contain knowledge of the test alerts or their expected results.

• Neither the AI models nor the instructions may include or be trained on the test alerts or their expected results.

• Any AI model may be used that is not fine-tuned on this data set.

5.7. Guidance on use

The method applies beyond this process to other regulated, rule-driven work. The benchmark may be used as published to compare agentic platforms or AI models, with variations that model a specific bank, or as a template for benchmarks in related domains. Vendors may publish their results on this benchmark, provided they cite this article as its source and state the AI model, the platform and the implementation used.

Learn more

• Benchmark definition site: https://github.com/ThunkAI/aml-benchmark/

• Benchmark results on Thunk.AI: https://docs.thunk.ai/benchmarks/aml-benchmark-2026-10/results.html

• AI Reliability for Drug Safety Case Intake (October 2026): the ICSR benchmark

• AI Reliability for IT Service Management (February 2026): the ITSM benchmark

• Thunk.AI website: https://www.thunk.ai