The "HiFi" Benchmark for Reliable AI Automation
The "HiFi" Benchmark for Reliable AI Automation
AI Reliability for Customer Service Agents: the τ²-bench Retail Benchmark
AI Reliability for Customer Service Agents: the τ²-bench Retail Benchmark
Highlights
This benchmark measures how much an application platform adds to the reliability of an LLM on a public, widely used agent benchmark: the retail customer service domain of τ²-bench (tau2-bench), run on gpt-5.1, a capable but non-frontier LLM.
The platform made the same LLM markedly more consistent. On 69 held-out tasks, run four times each, gpt-5.1 on its own passed every trial of a task 32% of the time (pass^4 0.32). On Thunk.AI, with the modular design, it did so 49% of the time (pass^4 0.49), and its single-trial success rose from 0.63 to 0.73.
τ²-bench retail, 69 held-out tasks × 4 trials; details in section 5.
Prompting an LLM is the least reliable way to build an agent. Given the policy as a prompt and the store's tools, gpt-5.1 never solved 9 of the 69 tasks in any of its four trials, and solved only 22 every time. With the modular thunk, 5 were never solved and 34 always were.
A simple mapping to the platform helps; a structured one helps more. Moving the policy into a single-step thunk with policy-checked tools raised pass^4 from 0.32 to 0.43. Decomposing it into steps, typed state, code for the deterministic rules and AI tools for the judgments raised it to 0.49.
The fixable failures came from the specification, not from LLM weakness. The failures the platform removed traced to two causes: ambiguity in the written policy, and exact rules left for the LLM to apply in its head. Some failures remain that are plain LLM errors; no platform can remove those, but it can give the LLM fewer chances to make them.
The tuning itself was automated. Thunk.AI's Forward Deployed Agent derived the specification, built the thunks, ran the benchmark, analyzed every failure and built the fixes, over eight iterations. A person only steered. This is the iterative process every enterprise agent needs, and it can be almost entirely automated.
τ²-bench was not built by Thunk.AI: some of its expected results disagree with its own policy, and its customers are simulated by an LLM, so no implementation scores 100%. The measure that matters is the relative improvement over the same LLM, on the same tasks, under the same noise.
Resources:
Benchmark definition site
Benchmark results on Thunk.AI
Thunk.AI Website
Highlights
This benchmark measures how much an application platform adds to the reliability of an LLM on a public, widely used agent benchmark: the retail customer service domain of τ²-bench (tau2-bench), run on gpt-5.1, a capable but non-frontier LLM.
The platform made the same LLM markedly more consistent. On 69 held-out tasks, run four times each, gpt-5.1 on its own passed every trial of a task 32% of the time (pass^4 0.32). On Thunk.AI, with the modular design, it did so 49% of the time (pass^4 0.49), and its single-trial success rose from 0.63 to 0.73.
τ²-bench retail, 69 held-out tasks × 4 trials; details in section 5.
Prompting an LLM is the least reliable way to build an agent. Given the policy as a prompt and the store's tools, gpt-5.1 never solved 9 of the 69 tasks in any of its four trials, and solved only 22 every time. With the modular thunk, 5 were never solved and 34 always were.
A simple mapping to the platform helps; a structured one helps more. Moving the policy into a single-step thunk with policy-checked tools raised pass^4 from 0.32 to 0.43. Decomposing it into steps, typed state, code for the deterministic rules and AI tools for the judgments raised it to 0.49.
The fixable failures came from the specification, not from LLM weakness. The failures the platform removed traced to two causes: ambiguity in the written policy, and exact rules left for the LLM to apply in its head. Some failures remain that are plain LLM errors; no platform can remove those, but it can give the LLM fewer chances to make them.
The tuning itself was automated. Thunk.AI's Forward Deployed Agent derived the specification, built the thunks, ran the benchmark, analyzed every failure and built the fixes, over eight iterations. A person only steered. This is the iterative process every enterprise agent needs, and it can be almost entirely automated.
τ²-bench was not built by Thunk.AI: some of its expected results disagree with its own policy, and its customers are simulated by an LLM, so no implementation scores 100%. The measure that matters is the relative improvement over the same LLM, on the same tasks, under the same noise.
Resources:
Benchmark definition site
Benchmark results on Thunk.AI
Thunk.AI Website
This article describes how the benchmark was run, the three implementations compared, their results on the 30 iteration tasks and the 69 held-out tasks, and how the Forward Deployed Agent tuned the design.
1: Why this benchmark
Benchmarks like τ²-bench were built to show how LLMs compare on an enterprise-like workload. Each domain gives the LLM a written policy (a prompt) and a set of tools, then runs many randomized customer conversations that are expected to be handled under that policy. A simulated customer plays each scenario, and the conversation passes only if the store's database ends in exactly the expected state.
A central part of the measurement is consistency. τ²-bench reports pass^k: the probability that a task passes in all of k independent trials. A business process that works in three conversations out of four is not one an enterprise can rely on, and pass^k exposes exactly that gap.
In practice, this is not how agentic applications are built for the enterprise, or for any repetitive use. Sending a prompt to an LLM is the naive way to use agentic AI, and it minimizes consistency. Nothing analyzes the prompt in advance, finds where it is ambiguous, or separates its exact rules from its judgment calls, because a bare LLM has no application model beyond the prompt.
An application platform like Thunk.AI does have one: steps with defined inputs and outputs, typed state, code tools, AI tools with their own instructions and tools, and checks on what the AI records. This benchmark sets out to measure how much reliability that application model adds, on the same tasks, the same simulated customers and the same LLM.
What the numbers mean. τ²-bench was not created by Thunk.AI, and it is not a perfect yardstick. In places its expected results disagree with its own written policy, and its customers are played by an LLM, which introduces inconsistencies of its own: a simulated customer may misremember its scenario, stop early, or answer differently from one trial to the next. No implementation can score 100% on it. What matters here is the relative improvement over the same LLM on its own, measured on the same tasks under the same noise.
This is the latest in Thunk.AI's "HiFi" series, which measures AI agents doing enterprise business processes. Earlier benchmarks in the series covered document workflows, IT service management, drug safety case intake and AML alert triage. Unlike those, this one uses a public benchmark that Thunk.AI did not design.
2: The retail customer service domain
In the retail domain, the agent is an online store's customer service representative. It must authenticate each customer by email, or by name and zip code, and can then help them with their own orders and profile. The policy allows six kinds of action, each with its own rules:
Action | Allowed when | Key rules |
|---|---|---|
Cancel an order | The order is pending | Whole order only; one of two accepted reasons |
Modify an order's address or payment | The order is pending | Payment: a single method, different from the original; a gift card must cover the amount |
Modify an order's items | The order is pending | Same product, different options; once per order, after which nothing else can change |
Return items | The order is delivered | Refund to the original method or a gift card |
Exchange items | The order is delivered | Same product, different options; a return and an exchange exclude each other |
Change the default address | Always, once authenticated | Any complete address |
The agent must list the details of every change and get an explicit yes before making it, must not make up information or procedures, and must transfer to a human agent "if and only if the request cannot be handled within the scope of your actions".
The work is exacting. Customers describe what they want loosely ("the cheapest one in blue", "the same colour as my other bottle", "my New York address, it's on my other order"), change their minds, combine several requests, and sometimes misremember their own orders. Each conversation passes only if the final database matches the expected one exactly: the right items, the right new variants, the right payment method, and no extra changes.
Scoring. Every task is run k times; pass^1 is the average success rate over all conversations, and pass^4 is the probability that all four trials of a task pass. The customer is simulated by gpt-4.1 at temperature 0, as in the public benchmark. The agent LLM is gpt-5.1 throughout.
3: Benchmark methodology
3.1. Three implementations
All three use the same LLM, gpt-5.1 with thinking off, temperature 0 and top_p 0, and the same policy, store tools and simulated customers. The standalone LLM is also reported at the benchmark's default settings, for reference.
Implementation | What it is |
|---|---|
Standalone gpt-5.1 | τ²-bench's own reference agent: the policy as the system prompt, the store's tools, one LLM call per turn |
Single-step thunk (v8) | One chat step holding the whole process, with the policy's write rules moved into checked code tools and a typed request ledger (Appendix A) |
Modular thunk (v9) | Two steps bound by typed properties, one AI tool per kind of request, typed read tools, code for every deterministic rule, and a separate judge for transfers (Appendix B). The iteration set reports its v9g version; the held-out set its final version, v9h |
The modular thunk is the design best suited to an enterprise. Each part is small enough to test on its own, each AI tool sees only the tools and policy it needs, every write goes through a checked tool that cites the rule it enforces, and every decision is recorded in typed state that can be audited.
3.2. Data sets
The retail domain has 114 tasks: a 40-task test split and a 74-task train split.
The iteration set: 30 tasks from the test split. Every design change was made by reading failed conversations on these tasks. Ten further test tasks were used to debug the first modular build and are left out of the comparison.
The held-out set: 69 tasks from the train split. Nobody looked at these tasks, their conversations or their expected results while the thunks were built. Each implementation ran on them once, at the end. (The other 5 train tasks were used to test the connection between τ²-bench and Thunk.AI.)
Every task ran four times per implementation: 120 conversations per run on the iteration set and 276 on the held-out set.
3.3. The tuning was done by Thunk.AI's Forward Deployed Agent
The thunks were designed, built, tested and refined by Thunk.AI's Forward Deployed Agent, an AI agent that builds and tunes thunks through the platform's builder interface. It derived the specification from the policy, mapped it onto the platform, ran the benchmark, read every failed conversation, classified each failure, proposed and built the fixes, and ran the next iteration. A person steered at a high level: choosing between the alternatives it proposed, deciding the clarifications of the policy, and rejecting any fix that was specific to a test rather than general. Section 6 describes the loop. This matters because it is exactly the iterative process every enterprise application needs, and it can be almost entirely automated.
3.4. The rule every change followed
Every change came from the written policy or from a platform mechanism, never from knowledge of a particular task. An ambiguity was settled by a written clarification, decided by the process owner (the policy's author, in a real deployment) and recorded in the specification. An agent error was fixed by moving more of the work into a platform mechanism: a smaller step, a code check, a typed field. Every proposed fix was reviewed for being general, and any that only fitted the test at hand was dropped.
3.5. Evaluation harness
Thunk.AI built a bridge so that a thunk can be measured exactly as τ²-bench measures its own agent. Each simulated conversation is one Thunk.AI chat conversation. Every store tool call the thunk makes, including those made inside its code and AI tools, is relayed back through τ²-bench and executed there, so the benchmark's own evaluator replays and scores the trajectory unchanged.
4: Results on the 30 iteration tasks
On the tasks it was tuned on, the modular thunk raised pass^1 from 0.63 to 0.83 and more than doubled pass^4, from 0.27 to 0.60. The single-step thunk captured most of the single-trial gain; the modular design added consistency, solving 18 of the 30 tasks in every trial against 8 for the LLM on its own.
τ²-bench retail, 30 iteration tasks × 4 trials. Standalone rerun at temperature 0; single-step thunk = v8, modular thunk = v9g.
These are the tasks the design was refined on, so they show what the process can reach, not what to expect on new work. Section 5 gives the results on tasks it never saw.
5: Results on the 69 held-out tasks
On tasks never seen during development, the modular thunk beat the standalone LLM at every level of consistency: pass^1 0.73 against 0.63, and pass^4 0.49 against 0.32, a 53% relative gain in the share of tasks handled correctly every time.
τ²-bench retail, 69 held-out train-split tasks × 4 trials (276 conversations per implementation). Standalone at temperature 0; single-step thunk = v8, modular thunk = v9h.
The gap widens as k grows. Each extra trial a task must pass costs the standalone LLM more than it costs the thunk, which is the signature of a more consistent system rather than a more capable one. The modular design nearly halved the tasks never solved (9 to 5) and moved 12 more tasks to always solved (22 to 34). The single-step thunk captured most of the single-trial gain; the modular design added the larger part of the consistency gain, and is the design an enterprise can test and audit piece by piece.
Every trial of every task, in every configuration, is listed in the detailed results.
Two cautions. With 276 conversations per run, pass^1 differences of about ±0.05 are within run-to-run noise, so the modular thunk's pass^1 lead over the single-step thunk (0.04) is suggestive, not conclusive; its pass^4 lead (0.06) and its halving of never-solved tasks are clearer. And the held-out gains are smaller than on the iteration set (pass^1 0.83 there), the expected cost of measuring on tasks the design never saw, and the reason these are the figures to quote.
The standalone LLM at two settings. Every comparison in this report uses the standalone LLM at the thunk's settings (thinking off, temperature 0, top_p 0). It was also run at the benchmark's default settings (temperature 1.0), on both task sets. Temperature barely moved it, and neither setting comes close to the thunk. On the 30 iteration tasks its pass^4 was 0.37 at temperature 1.0 and 0.27 at temperature 0, but that gap is 3 tasks out of 30 and within run-to-run noise (a resampling test puts a gap that large at about a 1-in-4 chance), and it did not recur here.
Standalone gpt-5.1 settings | pass^1 | pass^2 | pass^3 | pass^4 | Tasks 4 of 4 | Tasks 0 of 4 |
|---|---|---|---|---|---|---|
Benchmark defaults (temperature 1.0) | 0.61 | 0.46 | 0.38 | 0.33 | 23 | 10 |
Thinking off, temperature 0, top_p 0 | 0.63 | 0.48 | 0.38 | 0.32 | 22 | 9 |
6: Iterative tuning by the Forward Deployed Agent
The Forward Deployed Agent built the thunk in eight iterations, each one a full turn of the loop below. The person steering it never edited the thunk: they chose among the options it proposed, decided how to clarify the policy, and turned down fixes that only fitted a test.
the tuning loop · 5 automated stages, human steering, one held-out validation
Each turn of the loop ran the iteration set, read every failed conversation and classified it (a misreading of the policy, an ambiguity it leaves open, a mechanical fault, an agent error, or simulator noise), then built the fixes that the policy or the platform justified. The chart shows the result of each iteration.
τ²-bench retail, 30 iteration tasks × 4 trials. Standalone rerun at temperature 0; v4, v7 and v8 ran on the 40-task test split and are re-scored here on the same 30 tasks.
Version | What changed |
|---|---|
Standalone | τ²-bench's own agent: the policy as a prompt |
v4 | The policy in one Thunk.AI step, with a few checked tools |
v7 | A specification derived sentence by sentence from the policy; every write rule enforced in a code tool; the customer's choices left open |
v8 | Rules that span actions (an item change ends all other changes to an order; a return excludes an exchange), clarifications of the policy, an AI guard on transfers |
v9d | The modular redesign: two steps, one AI tool per kind of request, typed state, query-based variant search |
v9e | Constraints at the module boundaries: identifiers normalized, AI tools that cannot fail their caller, a list type for requests covering several orders |
v9f | Transfer judged separately from the transfer itself; ranked, typed search criteria; orders checked against the customer's description; addresses on file |
v9g | The calling step reports each result and moves on to the next request; stated conditions reported |
Modularity needs constraints at its boundaries. The first modular build (v9d) scored below the single step it replaced. Breaking the work into smaller units has real benefits, but it also encapsulates information at each boundary, which removes the LLM's chance to fix up an error later. In the single step, an order id typed as "W5061109" without its "#" would fail a lookup, and the LLM could see the error, look back at the conversation and try again. Inside the modular design, the same id passed into an AI tool failed there; the tool ended the step, the customer saw "There was an error handling your message", and nothing could go back to correct it. Modularity therefore has to come with more constraints and checks that make sure each module is set up correctly: identifiers normalized and typed at every tool boundary, multi-valued facts as lists, and AI tools that always return an outcome, even when they could not finish. With those in place (v9e), the modular design overtook the single step on pass^4, and two further iterations (v9f, v9g) brought pass^1 to 0.83 to 0.86.
The final version, v9h, revised one clarification (CL-1, Appendix B) after the v9g review and was measured only on the held-out set.
7: What made the difference
The failures the platform could fix traced to two causes, neither of them a basic weakness of the LLM: the policy was ambiguous, or an exact rule was left for the LLM to apply in its head. Some failures remain that are plain LLM errors, such as a customer's request misread or a step skipped. No platform can remove those, but it can leave the LLM fewer chances to make them, by giving it smaller, well-defined jobs and checking what it records.
7.1. Ambiguity in the specification
The retail policy is short and reads clearly, but it leaves real decisions open, and in places it disagrees with the benchmark's own tool descriptions. An LLM reading it fresh in every conversation resolves each gap differently from one conversation to the next, which is exactly what lowers pass^k. The analysis found and settled these, among others:
Ambiguity | Decision recorded | Mechanism |
|---|---|---|
The transfer tool says "transfer if the user asks"; the policy says "if and only if the request cannot be handled" | The policy wins over a tool's own description | A transfer is judged, then made, by separate tools |
Is a request the policy denies "one that cannot be handled"? | A denied, insisted-on or escalated request is still in scope; it is handled by being denied | The transfer judge reads the customer's own words, not the agent's reasons |
What happens to options the customer did not mention in an item change? | Variants that meet everything asked for are all shown, the one keeping current options marked; the customer chooses (CL-1) | A query tool returns every qualifying variant |
A request outside the agent's actions | Decline it plainly first; transfer only if the customer still wants it (CL-2) | The transfer tool requires that the customer was told |
May the agent suggest actions? | Never offer an action the customer did not request (CL-3) | Every write must cite a request the customer made |
A payment method named for a price difference | A parameter of that change, not a separate payment change | The request ledger |
In the first full review (v7), 12 of 40 failed conversations were misreadings of the policy that the first specification had missed, all traced to gaps like these.
7.2. Deterministic logic mixed with AI logic
The policy mixes judgment (what does the customer want?) with exact rules (is the order pending? does this gift card cover the difference? is this variant the cheapest?). Left in the prompt, the LLM applies the exact rules unreliably. In the review of the first modular build (v9d), the 40 failed conversations broke down like this:
Cause | Failed conversations | What the platform does now |
|---|---|---|
Mechanical: identifiers typed without their "#", a failing AI tool ending the step, several orders in one text field | 11 | Identifiers normalized by every tool; AI tools return an outcome instead of failing; a list type for multi-valued facts |
A ranked or referenced requirement ("the cheapest", "the highest resolution", "the same colour as my other one") worked out by the LLM | 9 | A query tool computes it in code: ranked criteria, a value order the LLM supplies, the current item compared, referenced items matched |
A transfer allowed for an in-scope request (the old guard approved 19 of 19 requests it saw) | 7 | Judging split from acting: an AI tool decides, a code tool transfers only on its decision |
A payment method recorded as a request of its own | 5 | Typed request ledger and a recorded clarification |
An order or item not checked against the customer's description | 3 | The order lookup itself reports products the customer named that are not on the order, and which orders have them |
A capability the specification named but the build lacked (addresses on other orders) | 2 | A typed address lookup |
Simulator or benchmark noise | 4 | None |
The pattern held across every round: once a rule was moved out of the prompt and into code, a type or a checked tool, the failures it caused stopped. The AI was left with what only AI can do: understand the customer, map their words onto the query, and hold the conversation.
8: Takeaways
This benchmark ran a non-frontier LLM on a workload with inherent ambiguity, so the absolute numbers are modest. The public τ²-bench leaderboard shows the latest frontier LLMs with much stronger baseline results, at 85% to 88% pass^1 across its domains (as of October 2026). What this benchmark shows is different: on the same LLM, the same tasks and the same simulated customers, an enterprise automation platform markedly improves reliability, by means that apply to any LLM.
Three lessons carry beyond this benchmark:
Most fixable failures start in the specification. A written policy that reads clearly still leaves decisions open, and an LLM resolves them differently from one run to the next. A platform that analyzes the intent, surfaces each ambiguity and records a decision on it turns inconsistency into a rule.
Exact rules belong in exact mechanisms. When deterministic logic is extracted into code tools and the type system, and constraints are enforced on what the AI records, the LLM is left with the judgment it is good at, and the rules are applied the same way every time. Modularity multiplies the benefit, but only with constraints and checks at every module boundary.
The tuning loop can be automated. Deriving the specification, building, testing and analyzing every failure was done by the Forward Deployed Agent, with a person steering. That makes this level of rigor practical for every enterprise agent, not only for a benchmark.
Summary of key findings
On 69 held-out tasks, gpt-5.1 on Thunk.AI (modular thunk) reached pass^1 0.73 and pass^4 0.49, against 0.63 and 0.32 on its own at the same settings.
The tasks it never solved fell from 9 to 5; the tasks it always solved rose from 22 to 34.
A single-step thunk captured most of the single-trial gain; the modular thunk added most of the consistency gain.
The bare LLM's results barely changed between temperature 1.0 and 0: sampling settings do not buy consistency; structure does.
The gains came from policy clarifications and platform mechanisms, not from prompting: every change restates or settles the written policy, and none is specific to a task.
Notes on method
Agent LLM gpt-5.1 with thinking off, temperature 0 and top_p 0 in every comparison; the standalone LLM is also reported at the benchmark's default settings. The simulated customer is gpt-4.1 at temperature 0. Each held-out run was made once, after development ended.
Every task ran four trials. A turn on which the thunk sent no reply was scored as an empty reply, and failed conversations were never restarted, so no failure was hidden.
The iteration set is the 30 test-split tasks the modular design was developed on. Earlier designs were developed on the whole 40-task test split; their scores are re-computed on the same 30.
Where the benchmark's expected result disagreed with a clarification grounded in the policy, the clarification stood: the thunk was not tuned to pass such tasks (for example, a task whose expected result transfers a customer who insists on a refund method the policy forbids).
Cost and latency were not measured in this round.
Appendix A: The single-step thunk (v8)
The single-step thunk (v8) keeps the shape of the original prompt: one chat step, "Serve the retail customer", holds the whole conversation. Its directions are a numbered process followed by the full retail policy, verbatim. What the platform adds is around that step.
Typed state.
AuthenticatedUserId, recorded only from a successful lookup, andCustomerRequests, a list of aCustomerRequeststructure type (kind, order, the customer's own words, status). Every change must cite an open request of the matching kind.Checked write tools. Six code tools wrap the store's writes:
cancel_order_checked,change_address,change_item_options,change_order_payment,return_items_checked, and the transfer. Each enforces the policy's rules in code (pending or delivered status, allowed reasons, a gift card covering the amount, once per order, a return excluding an exchange, an item change ending all other changes) and runs in two calls: a preview, then the write after the customer's explicit yes. The raw write tools are disabled in the step.Read helpers.
find_orders,search_variants,product_variant_summaryandfind_customer_addressescompute what the LLM would otherwise work out from raw records.A transfer guard.
transfer_to_human_checked, an AI tool, decides whether a request is outside the agent's actions before transferring.Enforced conversation. All messages go through
message_customer, which sends and waits for the reply; the step cannot finish itself, so it cannot end the conversation early.
v8 was built from a specification derived from the policy, sentence by sentence, with three rounds of clarifications from reviewing every failed conversation on the test split.
Appendix B: The modular thunk (v9h)
v9h decomposes the same policy into parts that each do one job, share context only through typed properties, and can be tested on their own.
Part | Kind | What it does |
|---|---|---|
Authenticate the customer | Step (New → Authenticated) | Only the two lookups, messaging and the transfer; must record |
Serve the customer's requests | Step (Authenticated → Done) | Records requests, identifies orders, routes each request to its AI tool, reports results and moves on; no write tools at all |
| AI tools | Each owns one kind of request with the customer, carries only its own section of the policy and its own tools, and returns a typed outcome (done, not allowed, withdrawn, blocked by another request, could not complete) |
| Code tools | Every write rule in code, preview then write on an explicit yes; each refuses a write that does not cite an open request the customer made |
| Code tools | Typed reads returning structure types ( |
| Code tool | A query over a product's variants: requested options, options copied from a referenced item, ranked criteria with an LLM-supplied value order, price limits, the current item compared but never offered |
| AI tool + code tool | The judge decides from the customer's own words and has no tools; the code tool transfers only on its decision |
| Typed properties | A constrained routing key; the verified user; a list of |
Clarifications log. Each entry settles a point the policy leaves open or contradicts, and is recorded in the specification with the rule it rests on.
Id | Clarification | Grounded in |
|---|---|---|
Precedence | Where the policy and a tool's description disagree, the policy wins | The policy is the source of truth |
R6' | A request that is one of the agent's actions stays in scope whatever happened to it; insisting or asking for a human does not change it | "Transfer if and only if the request cannot be handled within the scope of your actions" |
CL-1 | Stated requirements and selections are query constraints; when several variants meet them, all are shown, the one keeping current options marked, and the customer chooses | "The user must provide" the choice; the policy leaves unmentioned options open |
CL-2 | A request outside the agent's actions is declined plainly first; transfer only if the customer still wants it | The transfer rule and "no procedures not provided by the tools" |
CL-3 | Never offer an action the customer did not request | "No subjective recommendations" |
R5' | A payment method named for a difference or a refund is part of that request | "The user must provide a payment method" for each change |
R17 | Identifying the order is the agent's job; an order the customer names is checked against what they describe | "Help the user look up order id" |
R18 | The current item is never offered as its own replacement, but is compared | "A new item of the same product but of different product option" |
R19 | Every price, comparison and completed action stated to the customer comes from a tool result | "Do not make up any information" |
Appendix C: Benchmark details
C.1. The benchmark. τ²-bench is an open benchmark of conversational agents in dual-control environments, published by Sierra Research (tau2-bench on GitHub). This report uses its retail domain unchanged: the policy, the store database, the tools, the tasks, the user simulator and the evaluator.
C.2. Data sets used.
Set | Split | Tasks | Use |
|---|---|---|---|
Iteration set | test | 30 | Every design change; results in section 4 |
Design debugging | test | 10 | Debugging the first modular build; not reported |
Held-out set | train | 69 | Measured once per implementation at the end; results in section 5 |
Harness testing | train | 5 | Testing the connection between τ²-bench and Thunk.AI |
C.3. Rules followed for a valid implementation.
The policy, the tools and the tasks were not changed. The policy may be restated, divided among steps and tools, and clarified where it is open, provided every clarification is recorded and grounded in the policy's own words.
Tools may apply the policy's rules; they must not contain knowledge of any task or its expected result.
No instruction, example or tool description may quote or paraphrase a task. Examples in instructions use neutral cases.
A clarification that disagreed with a task's expected result was kept, and the task left to fail.
The held-out tasks were not examined until every implementation had run on them.
C.4. Guidance on use. The method applies to any agent built from a written policy: derive a specification, settle its ambiguities with the policy's owner, move its exact rules into code and types, and give the AI only the judgments that need it. Other LLMs can be run through the same harness to see how far the platform narrows the differences between them.
Learn more
τ²-bench: github.com/sierra-research/tau2-bench
Benchmark results on Thunk.AI: docs.thunk.ai/benchmarks/tau2-retail-benchmark-2026-10/results.html
AI Reliability for AML Alert Triage (October 2026): the AML benchmark
AI Reliability for Drug Safety Case Intake (October 2026): the ICSR benchmark
AI Reliability for IT Service Management (February 2026): the ITSM benchmark
Thunk.AI website: thunk.ai
This article describes how the benchmark was run, the three implementations compared, their results on the 30 iteration tasks and the 69 held-out tasks, and how the Forward Deployed Agent tuned the design.
1: Why this benchmark
Benchmarks like τ²-bench were built to show how LLMs compare on an enterprise-like workload. Each domain gives the LLM a written policy (a prompt) and a set of tools, then runs many randomized customer conversations that are expected to be handled under that policy. A simulated customer plays each scenario, and the conversation passes only if the store's database ends in exactly the expected state.
A central part of the measurement is consistency. τ²-bench reports pass^k: the probability that a task passes in all of k independent trials. A business process that works in three conversations out of four is not one an enterprise can rely on, and pass^k exposes exactly that gap.
In practice, this is not how agentic applications are built for the enterprise, or for any repetitive use. Sending a prompt to an LLM is the naive way to use agentic AI, and it minimizes consistency. Nothing analyzes the prompt in advance, finds where it is ambiguous, or separates its exact rules from its judgment calls, because a bare LLM has no application model beyond the prompt.
An application platform like Thunk.AI does have one: steps with defined inputs and outputs, typed state, code tools, AI tools with their own instructions and tools, and checks on what the AI records. This benchmark sets out to measure how much reliability that application model adds, on the same tasks, the same simulated customers and the same LLM.
What the numbers mean. τ²-bench was not created by Thunk.AI, and it is not a perfect yardstick. In places its expected results disagree with its own written policy, and its customers are played by an LLM, which introduces inconsistencies of its own: a simulated customer may misremember its scenario, stop early, or answer differently from one trial to the next. No implementation can score 100% on it. What matters here is the relative improvement over the same LLM on its own, measured on the same tasks under the same noise.
This is the latest in Thunk.AI's "HiFi" series, which measures AI agents doing enterprise business processes. Earlier benchmarks in the series covered document workflows, IT service management, drug safety case intake and AML alert triage. Unlike those, this one uses a public benchmark that Thunk.AI did not design.
2: The retail customer service domain
In the retail domain, the agent is an online store's customer service representative. It must authenticate each customer by email, or by name and zip code, and can then help them with their own orders and profile. The policy allows six kinds of action, each with its own rules:
Action | Allowed when | Key rules |
|---|---|---|
Cancel an order | The order is pending | Whole order only; one of two accepted reasons |
Modify an order's address or payment | The order is pending | Payment: a single method, different from the original; a gift card must cover the amount |
Modify an order's items | The order is pending | Same product, different options; once per order, after which nothing else can change |
Return items | The order is delivered | Refund to the original method or a gift card |
Exchange items | The order is delivered | Same product, different options; a return and an exchange exclude each other |
Change the default address | Always, once authenticated | Any complete address |
The agent must list the details of every change and get an explicit yes before making it, must not make up information or procedures, and must transfer to a human agent "if and only if the request cannot be handled within the scope of your actions".
The work is exacting. Customers describe what they want loosely ("the cheapest one in blue", "the same colour as my other bottle", "my New York address, it's on my other order"), change their minds, combine several requests, and sometimes misremember their own orders. Each conversation passes only if the final database matches the expected one exactly: the right items, the right new variants, the right payment method, and no extra changes.
Scoring. Every task is run k times; pass^1 is the average success rate over all conversations, and pass^4 is the probability that all four trials of a task pass. The customer is simulated by gpt-4.1 at temperature 0, as in the public benchmark. The agent LLM is gpt-5.1 throughout.
3: Benchmark methodology
3.1. Three implementations
All three use the same LLM, gpt-5.1 with thinking off, temperature 0 and top_p 0, and the same policy, store tools and simulated customers. The standalone LLM is also reported at the benchmark's default settings, for reference.
Implementation | What it is |
|---|---|
Standalone gpt-5.1 | τ²-bench's own reference agent: the policy as the system prompt, the store's tools, one LLM call per turn |
Single-step thunk (v8) | One chat step holding the whole process, with the policy's write rules moved into checked code tools and a typed request ledger (Appendix A) |
Modular thunk (v9) | Two steps bound by typed properties, one AI tool per kind of request, typed read tools, code for every deterministic rule, and a separate judge for transfers (Appendix B). The iteration set reports its v9g version; the held-out set its final version, v9h |
The modular thunk is the design best suited to an enterprise. Each part is small enough to test on its own, each AI tool sees only the tools and policy it needs, every write goes through a checked tool that cites the rule it enforces, and every decision is recorded in typed state that can be audited.
3.2. Data sets
The retail domain has 114 tasks: a 40-task test split and a 74-task train split.
The iteration set: 30 tasks from the test split. Every design change was made by reading failed conversations on these tasks. Ten further test tasks were used to debug the first modular build and are left out of the comparison.
The held-out set: 69 tasks from the train split. Nobody looked at these tasks, their conversations or their expected results while the thunks were built. Each implementation ran on them once, at the end. (The other 5 train tasks were used to test the connection between τ²-bench and Thunk.AI.)
Every task ran four times per implementation: 120 conversations per run on the iteration set and 276 on the held-out set.
3.3. The tuning was done by Thunk.AI's Forward Deployed Agent
The thunks were designed, built, tested and refined by Thunk.AI's Forward Deployed Agent, an AI agent that builds and tunes thunks through the platform's builder interface. It derived the specification from the policy, mapped it onto the platform, ran the benchmark, read every failed conversation, classified each failure, proposed and built the fixes, and ran the next iteration. A person steered at a high level: choosing between the alternatives it proposed, deciding the clarifications of the policy, and rejecting any fix that was specific to a test rather than general. Section 6 describes the loop. This matters because it is exactly the iterative process every enterprise application needs, and it can be almost entirely automated.
3.4. The rule every change followed
Every change came from the written policy or from a platform mechanism, never from knowledge of a particular task. An ambiguity was settled by a written clarification, decided by the process owner (the policy's author, in a real deployment) and recorded in the specification. An agent error was fixed by moving more of the work into a platform mechanism: a smaller step, a code check, a typed field. Every proposed fix was reviewed for being general, and any that only fitted the test at hand was dropped.
3.5. Evaluation harness
Thunk.AI built a bridge so that a thunk can be measured exactly as τ²-bench measures its own agent. Each simulated conversation is one Thunk.AI chat conversation. Every store tool call the thunk makes, including those made inside its code and AI tools, is relayed back through τ²-bench and executed there, so the benchmark's own evaluator replays and scores the trajectory unchanged.
4: Results on the 30 iteration tasks
On the tasks it was tuned on, the modular thunk raised pass^1 from 0.63 to 0.83 and more than doubled pass^4, from 0.27 to 0.60. The single-step thunk captured most of the single-trial gain; the modular design added consistency, solving 18 of the 30 tasks in every trial against 8 for the LLM on its own.
τ²-bench retail, 30 iteration tasks × 4 trials. Standalone rerun at temperature 0; single-step thunk = v8, modular thunk = v9g.
These are the tasks the design was refined on, so they show what the process can reach, not what to expect on new work. Section 5 gives the results on tasks it never saw.
5: Results on the 69 held-out tasks
On tasks never seen during development, the modular thunk beat the standalone LLM at every level of consistency: pass^1 0.73 against 0.63, and pass^4 0.49 against 0.32, a 53% relative gain in the share of tasks handled correctly every time.
τ²-bench retail, 69 held-out train-split tasks × 4 trials (276 conversations per implementation). Standalone at temperature 0; single-step thunk = v8, modular thunk = v9h.
The gap widens as k grows. Each extra trial a task must pass costs the standalone LLM more than it costs the thunk, which is the signature of a more consistent system rather than a more capable one. The modular design nearly halved the tasks never solved (9 to 5) and moved 12 more tasks to always solved (22 to 34). The single-step thunk captured most of the single-trial gain; the modular design added the larger part of the consistency gain, and is the design an enterprise can test and audit piece by piece.
Every trial of every task, in every configuration, is listed in the detailed results.
Two cautions. With 276 conversations per run, pass^1 differences of about ±0.05 are within run-to-run noise, so the modular thunk's pass^1 lead over the single-step thunk (0.04) is suggestive, not conclusive; its pass^4 lead (0.06) and its halving of never-solved tasks are clearer. And the held-out gains are smaller than on the iteration set (pass^1 0.83 there), the expected cost of measuring on tasks the design never saw, and the reason these are the figures to quote.
The standalone LLM at two settings. Every comparison in this report uses the standalone LLM at the thunk's settings (thinking off, temperature 0, top_p 0). It was also run at the benchmark's default settings (temperature 1.0), on both task sets. Temperature barely moved it, and neither setting comes close to the thunk. On the 30 iteration tasks its pass^4 was 0.37 at temperature 1.0 and 0.27 at temperature 0, but that gap is 3 tasks out of 30 and within run-to-run noise (a resampling test puts a gap that large at about a 1-in-4 chance), and it did not recur here.
Standalone gpt-5.1 settings | pass^1 | pass^2 | pass^3 | pass^4 | Tasks 4 of 4 | Tasks 0 of 4 |
|---|---|---|---|---|---|---|
Benchmark defaults (temperature 1.0) | 0.61 | 0.46 | 0.38 | 0.33 | 23 | 10 |
Thinking off, temperature 0, top_p 0 | 0.63 | 0.48 | 0.38 | 0.32 | 22 | 9 |
6: Iterative tuning by the Forward Deployed Agent
The Forward Deployed Agent built the thunk in eight iterations, each one a full turn of the loop below. The person steering it never edited the thunk: they chose among the options it proposed, decided how to clarify the policy, and turned down fixes that only fitted a test.
the tuning loop · 5 automated stages, human steering, one held-out validation
Each turn of the loop ran the iteration set, read every failed conversation and classified it (a misreading of the policy, an ambiguity it leaves open, a mechanical fault, an agent error, or simulator noise), then built the fixes that the policy or the platform justified. The chart shows the result of each iteration.
τ²-bench retail, 30 iteration tasks × 4 trials. Standalone rerun at temperature 0; v4, v7 and v8 ran on the 40-task test split and are re-scored here on the same 30 tasks.
Version | What changed |
|---|---|
Standalone | τ²-bench's own agent: the policy as a prompt |
v4 | The policy in one Thunk.AI step, with a few checked tools |
v7 | A specification derived sentence by sentence from the policy; every write rule enforced in a code tool; the customer's choices left open |
v8 | Rules that span actions (an item change ends all other changes to an order; a return excludes an exchange), clarifications of the policy, an AI guard on transfers |
v9d | The modular redesign: two steps, one AI tool per kind of request, typed state, query-based variant search |
v9e | Constraints at the module boundaries: identifiers normalized, AI tools that cannot fail their caller, a list type for requests covering several orders |
v9f | Transfer judged separately from the transfer itself; ranked, typed search criteria; orders checked against the customer's description; addresses on file |
v9g | The calling step reports each result and moves on to the next request; stated conditions reported |
Modularity needs constraints at its boundaries. The first modular build (v9d) scored below the single step it replaced. Breaking the work into smaller units has real benefits, but it also encapsulates information at each boundary, which removes the LLM's chance to fix up an error later. In the single step, an order id typed as "W5061109" without its "#" would fail a lookup, and the LLM could see the error, look back at the conversation and try again. Inside the modular design, the same id passed into an AI tool failed there; the tool ended the step, the customer saw "There was an error handling your message", and nothing could go back to correct it. Modularity therefore has to come with more constraints and checks that make sure each module is set up correctly: identifiers normalized and typed at every tool boundary, multi-valued facts as lists, and AI tools that always return an outcome, even when they could not finish. With those in place (v9e), the modular design overtook the single step on pass^4, and two further iterations (v9f, v9g) brought pass^1 to 0.83 to 0.86.
The final version, v9h, revised one clarification (CL-1, Appendix B) after the v9g review and was measured only on the held-out set.
7: What made the difference
The failures the platform could fix traced to two causes, neither of them a basic weakness of the LLM: the policy was ambiguous, or an exact rule was left for the LLM to apply in its head. Some failures remain that are plain LLM errors, such as a customer's request misread or a step skipped. No platform can remove those, but it can leave the LLM fewer chances to make them, by giving it smaller, well-defined jobs and checking what it records.
7.1. Ambiguity in the specification
The retail policy is short and reads clearly, but it leaves real decisions open, and in places it disagrees with the benchmark's own tool descriptions. An LLM reading it fresh in every conversation resolves each gap differently from one conversation to the next, which is exactly what lowers pass^k. The analysis found and settled these, among others:
Ambiguity | Decision recorded | Mechanism |
|---|---|---|
The transfer tool says "transfer if the user asks"; the policy says "if and only if the request cannot be handled" | The policy wins over a tool's own description | A transfer is judged, then made, by separate tools |
Is a request the policy denies "one that cannot be handled"? | A denied, insisted-on or escalated request is still in scope; it is handled by being denied | The transfer judge reads the customer's own words, not the agent's reasons |
What happens to options the customer did not mention in an item change? | Variants that meet everything asked for are all shown, the one keeping current options marked; the customer chooses (CL-1) | A query tool returns every qualifying variant |
A request outside the agent's actions | Decline it plainly first; transfer only if the customer still wants it (CL-2) | The transfer tool requires that the customer was told |
May the agent suggest actions? | Never offer an action the customer did not request (CL-3) | Every write must cite a request the customer made |
A payment method named for a price difference | A parameter of that change, not a separate payment change | The request ledger |
In the first full review (v7), 12 of 40 failed conversations were misreadings of the policy that the first specification had missed, all traced to gaps like these.
7.2. Deterministic logic mixed with AI logic
The policy mixes judgment (what does the customer want?) with exact rules (is the order pending? does this gift card cover the difference? is this variant the cheapest?). Left in the prompt, the LLM applies the exact rules unreliably. In the review of the first modular build (v9d), the 40 failed conversations broke down like this:
Cause | Failed conversations | What the platform does now |
|---|---|---|
Mechanical: identifiers typed without their "#", a failing AI tool ending the step, several orders in one text field | 11 | Identifiers normalized by every tool; AI tools return an outcome instead of failing; a list type for multi-valued facts |
A ranked or referenced requirement ("the cheapest", "the highest resolution", "the same colour as my other one") worked out by the LLM | 9 | A query tool computes it in code: ranked criteria, a value order the LLM supplies, the current item compared, referenced items matched |
A transfer allowed for an in-scope request (the old guard approved 19 of 19 requests it saw) | 7 | Judging split from acting: an AI tool decides, a code tool transfers only on its decision |
A payment method recorded as a request of its own | 5 | Typed request ledger and a recorded clarification |
An order or item not checked against the customer's description | 3 | The order lookup itself reports products the customer named that are not on the order, and which orders have them |
A capability the specification named but the build lacked (addresses on other orders) | 2 | A typed address lookup |
Simulator or benchmark noise | 4 | None |
The pattern held across every round: once a rule was moved out of the prompt and into code, a type or a checked tool, the failures it caused stopped. The AI was left with what only AI can do: understand the customer, map their words onto the query, and hold the conversation.
8: Takeaways
This benchmark ran a non-frontier LLM on a workload with inherent ambiguity, so the absolute numbers are modest. The public τ²-bench leaderboard shows the latest frontier LLMs with much stronger baseline results, at 85% to 88% pass^1 across its domains (as of October 2026). What this benchmark shows is different: on the same LLM, the same tasks and the same simulated customers, an enterprise automation platform markedly improves reliability, by means that apply to any LLM.
Three lessons carry beyond this benchmark:
Most fixable failures start in the specification. A written policy that reads clearly still leaves decisions open, and an LLM resolves them differently from one run to the next. A platform that analyzes the intent, surfaces each ambiguity and records a decision on it turns inconsistency into a rule.
Exact rules belong in exact mechanisms. When deterministic logic is extracted into code tools and the type system, and constraints are enforced on what the AI records, the LLM is left with the judgment it is good at, and the rules are applied the same way every time. Modularity multiplies the benefit, but only with constraints and checks at every module boundary.
The tuning loop can be automated. Deriving the specification, building, testing and analyzing every failure was done by the Forward Deployed Agent, with a person steering. That makes this level of rigor practical for every enterprise agent, not only for a benchmark.
Summary of key findings
On 69 held-out tasks, gpt-5.1 on Thunk.AI (modular thunk) reached pass^1 0.73 and pass^4 0.49, against 0.63 and 0.32 on its own at the same settings.
The tasks it never solved fell from 9 to 5; the tasks it always solved rose from 22 to 34.
A single-step thunk captured most of the single-trial gain; the modular thunk added most of the consistency gain.
The bare LLM's results barely changed between temperature 1.0 and 0: sampling settings do not buy consistency; structure does.
The gains came from policy clarifications and platform mechanisms, not from prompting: every change restates or settles the written policy, and none is specific to a task.
Notes on method
Agent LLM gpt-5.1 with thinking off, temperature 0 and top_p 0 in every comparison; the standalone LLM is also reported at the benchmark's default settings. The simulated customer is gpt-4.1 at temperature 0. Each held-out run was made once, after development ended.
Every task ran four trials. A turn on which the thunk sent no reply was scored as an empty reply, and failed conversations were never restarted, so no failure was hidden.
The iteration set is the 30 test-split tasks the modular design was developed on. Earlier designs were developed on the whole 40-task test split; their scores are re-computed on the same 30.
Where the benchmark's expected result disagreed with a clarification grounded in the policy, the clarification stood: the thunk was not tuned to pass such tasks (for example, a task whose expected result transfers a customer who insists on a refund method the policy forbids).
Cost and latency were not measured in this round.
Appendix A: The single-step thunk (v8)
The single-step thunk (v8) keeps the shape of the original prompt: one chat step, "Serve the retail customer", holds the whole conversation. Its directions are a numbered process followed by the full retail policy, verbatim. What the platform adds is around that step.
Typed state.
AuthenticatedUserId, recorded only from a successful lookup, andCustomerRequests, a list of aCustomerRequeststructure type (kind, order, the customer's own words, status). Every change must cite an open request of the matching kind.Checked write tools. Six code tools wrap the store's writes:
cancel_order_checked,change_address,change_item_options,change_order_payment,return_items_checked, and the transfer. Each enforces the policy's rules in code (pending or delivered status, allowed reasons, a gift card covering the amount, once per order, a return excluding an exchange, an item change ending all other changes) and runs in two calls: a preview, then the write after the customer's explicit yes. The raw write tools are disabled in the step.Read helpers.
find_orders,search_variants,product_variant_summaryandfind_customer_addressescompute what the LLM would otherwise work out from raw records.A transfer guard.
transfer_to_human_checked, an AI tool, decides whether a request is outside the agent's actions before transferring.Enforced conversation. All messages go through
message_customer, which sends and waits for the reply; the step cannot finish itself, so it cannot end the conversation early.
v8 was built from a specification derived from the policy, sentence by sentence, with three rounds of clarifications from reviewing every failed conversation on the test split.
Appendix B: The modular thunk (v9h)
v9h decomposes the same policy into parts that each do one job, share context only through typed properties, and can be tested on their own.
Part | Kind | What it does |
|---|---|---|
Authenticate the customer | Step (New → Authenticated) | Only the two lookups, messaging and the transfer; must record |
Serve the customer's requests | Step (Authenticated → Done) | Records requests, identifies orders, routes each request to its AI tool, reports results and moves on; no write tools at all |
| AI tools | Each owns one kind of request with the customer, carries only its own section of the policy and its own tools, and returns a typed outcome (done, not allowed, withdrawn, blocked by another request, could not complete) |
| Code tools | Every write rule in code, preview then write on an explicit yes; each refuses a write that does not cite an open request the customer made |
| Code tools | Typed reads returning structure types ( |
| Code tool | A query over a product's variants: requested options, options copied from a referenced item, ranked criteria with an LLM-supplied value order, price limits, the current item compared but never offered |
| AI tool + code tool | The judge decides from the customer's own words and has no tools; the code tool transfers only on its decision |
| Typed properties | A constrained routing key; the verified user; a list of |
Clarifications log. Each entry settles a point the policy leaves open or contradicts, and is recorded in the specification with the rule it rests on.
Id | Clarification | Grounded in |
|---|---|---|
Precedence | Where the policy and a tool's description disagree, the policy wins | The policy is the source of truth |
R6' | A request that is one of the agent's actions stays in scope whatever happened to it; insisting or asking for a human does not change it | "Transfer if and only if the request cannot be handled within the scope of your actions" |
CL-1 | Stated requirements and selections are query constraints; when several variants meet them, all are shown, the one keeping current options marked, and the customer chooses | "The user must provide" the choice; the policy leaves unmentioned options open |
CL-2 | A request outside the agent's actions is declined plainly first; transfer only if the customer still wants it | The transfer rule and "no procedures not provided by the tools" |
CL-3 | Never offer an action the customer did not request | "No subjective recommendations" |
R5' | A payment method named for a difference or a refund is part of that request | "The user must provide a payment method" for each change |
R17 | Identifying the order is the agent's job; an order the customer names is checked against what they describe | "Help the user look up order id" |
R18 | The current item is never offered as its own replacement, but is compared | "A new item of the same product but of different product option" |
R19 | Every price, comparison and completed action stated to the customer comes from a tool result | "Do not make up any information" |
Appendix C: Benchmark details
C.1. The benchmark. τ²-bench is an open benchmark of conversational agents in dual-control environments, published by Sierra Research (tau2-bench on GitHub). This report uses its retail domain unchanged: the policy, the store database, the tools, the tasks, the user simulator and the evaluator.
C.2. Data sets used.
Set | Split | Tasks | Use |
|---|---|---|---|
Iteration set | test | 30 | Every design change; results in section 4 |
Design debugging | test | 10 | Debugging the first modular build; not reported |
Held-out set | train | 69 | Measured once per implementation at the end; results in section 5 |
Harness testing | train | 5 | Testing the connection between τ²-bench and Thunk.AI |
C.3. Rules followed for a valid implementation.
The policy, the tools and the tasks were not changed. The policy may be restated, divided among steps and tools, and clarified where it is open, provided every clarification is recorded and grounded in the policy's own words.
Tools may apply the policy's rules; they must not contain knowledge of any task or its expected result.
No instruction, example or tool description may quote or paraphrase a task. Examples in instructions use neutral cases.
A clarification that disagreed with a task's expected result was kept, and the task left to fail.
The held-out tasks were not examined until every implementation had run on them.
C.4. Guidance on use. The method applies to any agent built from a written policy: derive a specification, settle its ambiguities with the policy's owner, move its exact rules into code and types, and give the AI only the judgments that need it. Other LLMs can be run through the same harness to see how far the platform narrows the differences between them.
Learn more
τ²-bench: github.com/sierra-research/tau2-bench
Benchmark results on Thunk.AI: docs.thunk.ai/benchmarks/tau2-retail-benchmark-2026-10/results.html
AI Reliability for AML Alert Triage (October 2026): the AML benchmark
AI Reliability for Drug Safety Case Intake (October 2026): the ICSR benchmark
AI Reliability for IT Service Management (February 2026): the ITSM benchmark
Thunk.AI website: thunk.ai