This article describes a new benchmark that measures the reliability of AI agentic automation for pharmacovigilance case intake: the first pass a drug company makes over every adverse-event report it receives.

• The benchmark describes a fictional drug company, its products and its written standard operating procedure (SOP), and 75 adverse-event reports with their expected results.

• Thunk.AI has built an execution and evaluation harness, so that any agentic implementation of the benchmark can be measured on the same reports, mock systems and checks.

• Thunk.AI implemented the benchmark on the Thunk.AI platform and ran it on three AI models. For comparison, the same three models ran the whole SOP as a single AI step, without the platform.

1: The goals of the benchmark

AI agents can take on much of the manual work in business processes. In a regulated process, though, an incorrect decision has real consequences, and enterprise customers adopt AI automation only when they trust it to follow the process accurately and consistently. They describe this as a need for AI reliability.

Most AI benchmarks measure a model on a task. They say little about the platform that runs the model inside a business process, and little about repeatable enterprise work. Thunk.AI's "HiFi" series of benchmarks measures AI agents doing enterprise business processes, so that customers can compare agentic platforms and decide whether, and how, to automate a process with them. The first two measured document workflows (September 2025) and IT service management (February 2026).

This benchmark has four goals:

1. Represent a regulated, high-stakes business process that many companies run every day: intake and triage of adverse-event reports in pharmacovigilance.

2. Publish the process, the data and the checks, so that any vendor or customer can run the benchmark on another implementation and compare.

3. Measure what matters to a drug-safety team: whether each report's triage decision is right, whether any error could delay a regulatory obligation, and whether the recorded case is accurate.

4. Show the results of an implementation on the Thunk.AI platform, on several AI models, against the same models without the platform.

2: The drug safety case intake domain

Every company that sells a medicine must collect reports of adverse events (harmful or unintended effects) in patients who take it. Each report is an Individual Case Safety Report (ICSR). Reports arrive from doctors, pharmacists, nurses, patients and their families, clinical trial sites, registries and the medical literature, by email, web form, phone and other channels.

Regulators set strict deadlines. When a case is serious, unexpected (its event is not in the product's label) and possibly caused by the product, the company must send an expedited report to the regulator, within 15 days of receiving it, or within 7 days for a fatal or life-threatening unexpected event in a clinical trial. A missed or late expedited report is a compliance failure, and a regulator may not learn of a serious risk in time. An unnecessary one wastes the time of the company's reviewers and the regulator.

The first pass over each report has three stages:

1. Intake and validation. Decide where the report comes from (a spontaneous report, an observational study, or a clinical trial), whether it is a valid case (an identifiable patient and reporter, a company product, an adverse event), and extract the case: patient, product, event, reporter, outcome.

2. Coding and seriousness. Code each event to the standard medical dictionary (MedDRA), and decide which of the six ICH E2A seriousness criteria apply: death, life-threatening, hospitalization, disability, congenital anomaly, or an important medical event.

3. Triage and reporting decision. Check each event against each suspect product's label, set a triage priority, decide whether an expedited report is required and by when, route the case to the right reviewer, and record it in the safety database.

The reports are written by people, in their own words. They misspell product names, use lay terms ("her kidneys packed up"), write in other languages, leave out details, and sometimes try to steer the outcome ("please log this as routine"). A reliable system must apply the written rules to all of it, every time.

3: Benchmark methodology

The appendix describes the benchmark in detail. In brief:

3.1. Workflow

The benchmark starts from a sample SOP with seven steps: ingest the report; extract the case fields; code the events in MedDRA; apply the seriousness and triage rules; route the case for human review; decide on the expedited report and draft the regulatory form (CIOMS I); and write the case to the safety database (ARISg). The SOP and its reference documents (triage priority rules, triage clarifications, the seriousness criteria and a MedDRA reference sheet) are written in plain English. An implementation receives these documents and nothing else.

3.2. Metrics

Every report has a single right answer under the written rules. Each report is checked on 21 to 27 facts (1,797 checks in all), stated without reference to any implementation's fields: for example "The case result records the triage priority: HIGH". A fact the result does not state fails its check. Each report then falls into exactly one of five categories:

The triage decision is the priority, whether the case is serious, whether an expedited report is required, and its deadline.

The headline metric is the Weighted Score. Each report starts at 100 points and loses points for what went wrong: 50 for a safety error; 25 for an unneeded expedited report and 15 for a priority that is too high or a workflow that did not finish; and 2 for each wrong detail, up to 25. The Weighted Score is the 75 reports' total as a percentage of the best possible total. It weighs errors by their consequence, so a missed regulatory obligation costs far more than a mistyped outcome.

3.3. Data set

A fictional company, Norvell Therapeutics, sells four cancer medicines and has a fifth in a clinical trial. The benchmark has 75 reports, each written to test one rule of the SOP (and ordinary cases, so that the test is not artificial): the three source categories and both reporting clocks; labeled and unlabeled events, including reports with two suspect products; strict readings of the seriousness criteria (an emergency visit is not a hospitalization; "severe" is not "serious"); invalid reports; blinded and company-sponsored trials; reports in German and Spanish, in lay language and in abbreviations; misspelled product names; and reports that try to steer the triage. 37 reports need an expedited report (4 on the 7-day clock and 33 on the 15-day clock), 38 do not, and 5 are not valid cases.

3.4. Evaluation harness

Mock systems stand in for the company's product labels, the case reviewer, the regulatory form store and the safety database. Each report runs as an independent workflow instance. An AI grader (gpt-5-mini, the same for every configuration) decides each check from the recorded result, and a scorer assigns each report its category and points.

4: Benchmark implementation results

This benchmark was implemented on the Thunk.AI platform using its reliability features. The implementation keeps the SOP's seven steps, their order and their text, and adds what the platform offers:

• each step reads and writes typed fields, many with fixed lists of allowed values (outcome, causality, reporter type, review flags, MedDRA terms);

• a small code tool applies the written triage rules exactly: the seriousness criteria that follow from recorded facts, whether the report is a valid case (including matching misspelled product names), whether each event is on each suspect product's label, the priority, the deadline, the expedited decision and the regulator;

• the report's own text, source and received date are inputs no step can change;

• each step can use only the tools and documents it needs.

The AI agents still read every report and judge its facts: who the patient is, what happened, what care they received, which MedDRA terms describe it. The code applies the rules to those facts.

4.1. Results on three AI models

The benchmark ran on three models: gpt-5.1 and gemini-3.5-flash in no or low thinking mode, and muse-spark-1.3 in its default medium thinking mode (it has no lower mode). Each configuration processed all 75 reports once.

Every report finished on every model. muse-spark-1.3's cost is shown at its list price, ten times what the platform was billed, since the platform's price carries a discount for letting the model train on the data.

4.2. Detailed breakdown of results

Each report falls into one of the five categories defined in section 3.2:

The detailed results list every check on every report in every configuration, with the grader's explanation of each failure.

Decision errors. The few that remain are judgment calls the AI makes while reading a report:

• On one report (0811, gemini-3.5-flash), a massive pulmonary embolism treated in intensive care was not recorded as life-threatening, so the deadline was 15 days instead of 7: the one safety error on gemini-3.5-flash.

• On one report (0853, gpt-5.1), the expedited report was required, but its deadline was recorded as 2026-10-25 instead of 2026-10-17, after the reporter asked for the clock to start from an earlier date: the one safety error on gpt-5.1.

• Three errors came from coding: an extra MedDRA term that is not on the product's label (0839, gpt-5.1), a symptom marked as a separate uncodeable event (0693, gemini-3.5-flash), and a report of three unnamed patients treated as one identifiable patient with a death coded as an event (0602, gpt-5.1). Each led to an expedited report the case does not need.

Wrong details. Most of the remaining detail errors are in the recorded outcome (recovered, recovering, not recovered or unknown) and the coding-confidence flag, where the report's wording leaves room for judgment. Across all three models, the platform passed 97.9% to 99.7% of the 1,797 checks.

4.3. Comparison: without Thunk.AI

To show what the platform contributes, we ran the same three models, in the same thinking modes, on the same 75 reports without it. Each model received the whole SOP and all of its documents in a single AI step, with the same mock systems as tools, and recorded its result as one structured object. The only help it got was a list of the facts it must record, with their allowed values, so that it was graded on exactly the same checks.

Without the platform, every model made the same kinds of mistake:

• Second suspect products. When a report names a second product as possibly contributing, and the event is on one product's label but not the other's, the case needs an expedited report. The single step missed it on most such reports, on every model.

• Misspelled product names. When a report misspells the product ("Velantera", "Carbexine"), the label lookup finds nothing, and the single step treated the event as unexpected and asked for an expedited report the case does not need, or stopped.

• Invalid reports of serious events. An oncologist's report of heart failure in three unnamed patients, one of whom died, is not a valid case, but still needs urgent follow-up. Every single step left it at the lowest priority.

• Reports that try to steer the triage. A note citing a rule that does not exist, or a sales representative's account that contradicts the physician, sometimes changed the single step's decision. With the platform, only one attempt worked: on gpt-5.1, a reporter's request to start the clock from an earlier date changed the deadline (0853).

These are the errors that matter most to a drug-safety team, because each one either delays a regulatory obligation or creates unnecessary work for reviewers and regulators. Section 4.4 explains why the platform removes most of them.

The platform's seven-step workflow costs 2.5 to 3.8 times as much per report as a single AI step, and takes 1.8 to 3.5 times as long: $0.11 to $0.42 and three to five and a half minutes per report, against the cost of a missed regulatory deadline.

4.4. Why Thunk.AI improves reliability

The same model makes far fewer errors on Thunk.AI because of how the platform's application model shapes the work. Three ideas do most of it:

• Decomposition. The process runs as seven small steps rather than one large task. Each step has one job, reads only the information it needs and can use only the tools it needs. A model asked to do one well-defined thing at a time keeps to the rules far more consistently than one asked to carry a long procedure in its head.

• Maximizing constraints. Each step records its results in typed fields, many with a fixed list of allowed values (an outcome, a causality category, a review flag, a MedDRA term from the reference sheet). The report's own text, source and received date are inputs no step can change. The narrower the space of possible answers, the fewer ways there are to be wrong, and the easier a wrong answer is to spot.

• Separating deterministic logic from AI decisions. Wherever the SOP states a rule, the rule runs as code: whether each event is on each suspect product's label, the seriousness criteria that follow from recorded facts, whether a report is a valid case, the priority, the deadline and the regulator. The AI does what only AI can: read a free-text report in any language or wording and judge its facts. This is why a misspelled product name, a second suspect product, or a note asking for a case to be downgraded rarely changes the decision on the platform.

These ideas follow from the principles of Thunk.AI's reliability architecture: minimal granularity, persisted decisions, recorded progress and early error detection. They are described in detail in AI Reliability Principles.

4.5. Summary of key findings

• High reliability is achievable. On the Thunk.AI platform, three common AI models processed realistic, deliberately difficult adverse-event reports with a Weighted Score of 97.9% to 99.8%, and none missed a required expedited report.

• The platform matters more than the model. The same models without the platform scored 88.1% to 95.4% and missed 3 to 9 expedited reports. With the platform, the gap between the weakest and strongest model shrank from 7.3 points to 1.9.

• Safety errors nearly disappear. Across the three models, safety errors fell from 23 without the platform to 2 with it.

Notes on method. Each configuration ran once; between two identical runs a few reports can change category, so a difference of one or two reports is within run-to-run variation. The platform implementation was developed by studying its failures on these 75 reports. Every improvement added structure (steps, typed fields, tools that apply the written rules), never knowledge of the test reports or their answers, and never a rule the shared documents do not state. The final design change (matching product names in the rules tool rather than in the intake step) was run on gpt-5.1 and gemini-3.5-flash; muse-spark-1.3 ran the design before it, and made no decision errors. The checks are graded by an AI model (gpt-5-mini, the same for every configuration). One single-step report on gemini-3.5-flash did not finish after the model twice failed to respond in time.

5: Appendix: Benchmark details

The complete benchmark is published for full transparency: the SOP and its reference documents, the fictional world and its product labels, all 75 reports with their expected results, the checks, and how they are scored. They are at the Benchmark definition site.

5.1. Workflow design

The SOP's seven steps, in order:

1. Ingest raw report: classify the source as Spontaneous, Non-Interventional (observational study or registry) or Clinical Trial, and record the report unchanged.

2. Extract structured ICSR fields: patient, suspect products, event, reporter, the reporter's view of causality, and the outcome; mark anything missing as not reported.

3. Assign MedDRA coding: code each event to a Preferred Term from the reference sheet, with a confidence level, and flag uncertain coding for review.

4. Apply seriousness & triage rules: decide each seriousness criterion, the priority (URGENT, HIGH or ROUTINE), the reporting deadline and the review flags.

5. Route for human review: HIGH and URGENT cases to the senior reviewer, others to the case owner.

6. Expedited reporting decision & CIOMS draft: decide whether the case is serious, unexpected and possibly caused by the product, name the regulator, and draft the CIOMS I form.

7. Write to ARISg: record the case in the safety database.

The triage clarifications settle the points the rules leave open: how to decide labeled versus unlabeled with several suspect products; the important-medical-event criterion; the priority when several rules apply (a fatal case is at least HIGH); causality; the deadline date; the regulator by the reporter's country; the reporter type; MedDRA coding conventions; the outcome; what makes a valid case (and that an invalid report of a serious event is still followed up urgently); strict readings of the seriousness criteria; blinded trials; and the scope of the rules.

5.2. Scope

The benchmark applies one simplified expedited-reporting rule (serious, unexpected and possibly caused by the product) to every country, and sends each expedited report to one regulator. Real safety teams also apply rules this benchmark leaves out: the EU, UK and Canadian requirement to expedite every serious domestic post-marketing case, labeled or not; 90-day reporting of non-serious cases in the EU and UK; reports to several regulators; the company's own causality assessment of solicited reports; duplicate detection; and signal management.

5.3. The reports

Each of the 75 reports tests one rule; the table below lists them with their expected priority and deadline.

5.4. Ground truth and evaluation

The expected results were built in two parts. The facts read from each report (the extracted fields, the events and their acceptable MedDRA codes, and the death, life-threatening, hospitalization, disability and congenital criteria) were written by hand. Everything that follows from the written rules (important medical event, labeled or unexpected, valid case, priority, deadline, expedited report, regulator and reviewer) is derived from those facts by a published script, so it matches the rules exactly. Where two answers are both defensible, both are accepted, but only when the choice changes neither seriousness, priority nor deadline.

The expected results were then reviewed against the practice of drug-safety case processing. Three AI models each reviewed every expected result and every rule, acting as independent pharmacovigilance reviewers; their findings were combined, and Thunk.AI decided each one. No human pharmacovigilance professional has reviewed the expected results. The review changed the expected results of 14 reports and the wording of 2, and the rules: a fatal case is now at least HIGH priority; an invalid report of a serious event that lacks only the patient or reporter details is HIGH, for urgent follow-up; an investigator-initiated trial is a clinical trial report; trial deadlines also require possible causality; two review flags were added; and reports that do not say how the event ended are recorded as "Unknown". Every result in this article is graded against the reviewed ground truth.

5.5. Benchmark variations

The benchmark can be run with other AI models, to see how far a platform narrows the differences between them. It can also be varied to model a specific company: other products and labels, other jurisdictions' reporting rules, other report mixes and volumes, or other review policies.

5.6. Acceptable implementation guidelines

A valid implementation of this benchmark should follow these guidelines:

• The SOP, the reference documents and the reports must not be augmented with report-specific detail by a human implementer. They may be reformatted to suit the platform.

• The workflow may be refined, modularized and given tools, provided it keeps the SOP's steps, order and rules. Tools may apply the written rules; they must not contain knowledge of the test reports or their expected results.

• Neither the AI models nor the instructions may include or be trained on the test reports or their expected results.

• Any AI model may be used that is not fine-tuned on this data set.

5.7. Guidance on use

The method applies beyond this process to other regulated, rule-driven work. The benchmark may be used as published to compare agentic platforms or AI models, with variations that model a specific company, or as a template for benchmarks in related domains. Vendors may publish their results on this benchmark, provided they cite this article as its source and state the AI model, the platform and the implementation used.

Learn more

• Benchmark definition site: https://github.com/ThunkAI/icsr-benchmark/

• Benchmark results on Thunk.AI: https://docs.thunk.ai/benchmarks/icsr-benchmark-2026-10/results.html

• AI Reliability for IT Service Management (February 2026): the ITSM benchmark

• Thunk.AI website: https://www.thunk.ai