AI Reliability in Practice
AI Reliability in Practice
Where Unreliable AI Behavior Comes From
Where Unreliable AI Behavior Comes From
Highlights
When an AI-automated workflow gets a case wrong, the instinct is to blame the model and try a stronger one. Agentic workflows fail in five different ways, and only one of them improves with a better model.
Instruction errors. The procedure is ambiguous, incomplete or contradicts the systems it describes, so the model has to guess.
Configuration errors. The workflow is built in a way that allows mistakes: a step can overwrite another step's decision, has tools it should not have, or enforces a rule the procedure does not contain.
LLM errors. The model itself makes a mistake: it misapplies an exact rule, copies a value wrongly, invents a missing value, or acts when it should ask. This is the only kind a better model reduces, and even here it does not remove them.
Orchestration errors. The loop that runs the agent lets it stop early, finish with the work undone, or lose track of what it has done.
Tool errors. A tool returns the wrong data or too much of it, accepts malformed input, fails, or repeats an action on retry.
Thunk.AI addresses each kind in a different layer of the platform: the App Model, the Authoring Environment and the Execution Runtime.
Highlights
When an AI-automated workflow gets a case wrong, the instinct is to blame the model and try a stronger one. Agentic workflows fail in five different ways, and only one of them improves with a better model.
Instruction errors. The procedure is ambiguous, incomplete or contradicts the systems it describes, so the model has to guess.
Configuration errors. The workflow is built in a way that allows mistakes: a step can overwrite another step's decision, has tools it should not have, or enforces a rule the procedure does not contain.
LLM errors. The model itself makes a mistake: it misapplies an exact rule, copies a value wrongly, invents a missing value, or acts when it should ask. This is the only kind a better model reduces, and even here it does not remove them.
Orchestration errors. The loop that runs the agent lets it stop early, finish with the work undone, or lose track of what it has done.
Tool errors. A tool returns the wrong data or too much of it, accepts malformed input, fails, or repeats an action on retry.
Thunk.AI addresses each kind in a different layer of the platform: the App Model, the Authoring Environment and the Execution Runtime.
This article describes the five ways an AI-automated workflow fails, with examples from the HiFi benchmarks, and which layer of the Thunk.AI platform prevents each. Its purpose is practical: when a process fails, the kind of failure decides the fix, and in four cases out of five the fix is not a different model.
1: How an agentic workflow runs
In an agentic workflow, an AI model decides at run time what to do next, instead of following a hard-coded sequence. At each stage the agent runs a loop: it reads its instructions, chooses a tool, calls it, reads the result, and decides what to do next, until the work is done. The tools reach the business's systems: a ticketing system, a document reader, a finance system. This flexibility is what lets an agent handle varied inputs and recover from the unexpected. It is also what makes its behavior non-deterministic, and gives each part of the loop its own way to fail.
2: Instruction errors: the procedure leaves the model to guess
The most common source of unreliable behavior is in the instructions themselves. A standard operating procedure written for trained employees leaves things unsaid, uses terms it does not define, and sometimes contradicts the systems it describes. A model given such a procedure does not know what is missing. It produces a plausible answer, and because the answer is a guess, it can differ from one run to the next.
In the τ²-bench retail work, the store's policy says a request may be transferred to a human agent "if and only if" it cannot be handled within the agent's actions, while the transfer tool's own description says to transfer when the user explicitly asks. The two sources disagree, and no model can know which one the business means. Instructions are ambiguous and incomplete explains why these gaps have to be found and settled at design time, in the Authoring Environment, and enforced in the App Model.
3: Configuration errors: the workflow is built to allow mistakes
A workflow can follow the procedure faithfully and still be configured in a way that lets the agent go wrong. In the first version of the AML alert triage thunk, every step could write every field, so the step that recorded the disposition could override the decision the reasoning step had reached. The rules tool was available in more steps than needed it, so a later step could call it again and record a different answer. Facts were recorded as free text, so values outside the rules, such as a risk rating of "High-ish", could appear.
Configuration errors also run the other way. Early versions of the τ²-bench thunk enforced rules the store's policy did not contain, such as defaulting the payment method for a refund when the policy says the customer must choose one. A workflow that is more constrained than the business intends is as wrong as one that is less.
These are design decisions, and they are caught at design time: in the Authoring Environment, where a design-time AI agent works with the designer and workflow review checks the design before it runs.
4: LLM errors: the model makes a mistake
Some failures are the model's own, even when the instructions are clear and the workflow is well built.
Misapplying exact rules. Procedures contain thresholds, windows, lists and counts. In the AML benchmark, models given the whole procedure as a single step counted two deposits as structuring, treated an 18-month-old report as "over 24 months", and rounded a 79% pass-through up to 80%. The write-up's field-level results show the pattern: the models gathered the facts well and made their mistakes when they had to apply many exact rules at once.
Copying values incorrectly. An agent that carries an id, an amount or a name from one place to another sometimes changes it on the way.
Inventing what is missing. When an output is required and the information is not in the input, a model tends to supply something plausible rather than say it does not know.
Misjudging its own scope. In the IT service management benchmark, GPT-4.1 on its own escalated only 1% of tickets to a person and took incorrect automated actions on 14%. In the AML benchmark, models on their own made between 9 and 40 unnecessary escalations each.
A better model makes fewer of these mistakes, but none of them disappear. The platform reduces them in two ways. The App Model takes away work the model does badly: exact rules move into code tools, facts become typed properties with fixed values, and judgment is confined to narrow questions. The Execution Runtime checks what the model does and feeds failures back for correction.
5: Orchestration errors: the loop lets the agent drift
The loop that runs the agent has its own failure modes. An agent can stop before the work is done and declare itself finished. In the AML benchmark, gpt-5-mini, given the procedure as a single step, finished 17 of 82 alerts without a usable result. An agent can lose track of what it has established, or a later stage can overturn an earlier one. An agent that answers in free text instead of structured actions behaves more variably from one run to the next.
These are prevented by the Execution Runtime, which runs the loop rather than leaving it to the model: the model can respond only with structured tool calls, progress is saved to typed state as it goes, every run must end with an explicit action, and the step's own rules still apply after the agent says it is done. How Thunk.AI runs an AI agent describes these mechanisms.
6: Tool errors: the tools give the agent the wrong footing
Tools are how an agent sees and changes the business's systems, and they fail in their own ways. A tool that returns a whole record when the step needs three fields invites the agent to use the wrong one. A tool with a loose input schema accepts malformed arguments. External systems time out or return errors. A tool with side effects, such as sending a message or filing a report, can repeat the action if a failed step is retried. Real data is also imperfect: the first HiFi benchmark deliberately introduced errors into both its invoices and its vendor reference data, because a process that only works on clean data will not work in production.
Tool errors are prevented in the App Model, by tools with strict schemas that return only what the step needs and that check their own inputs and refuse when a rule is not met, and in the Execution Runtime, which vets every call before it runs, gives each step only the tools it is allowed, and logs every call with its inputs and outputs.
7: Why a stronger model is not the answer
Four of the five kinds of failure have nothing to do with the model's capability. A stronger model does not resolve an ambiguity in the procedure, correct a step that is allowed to overwrite another, stop a loop that ends early, or fix a tool that returns the wrong data. The HiFi benchmarks show the consequence: on Thunk.AI, the gap between the best and worst models shrank from 22.6 points to 0.6 on AML alerts, and from 7.3 points to 1.9 on drug safety cases. What is in a standard operating procedure, and how each part is made reliable describes the design remedies in detail.
This article describes the five ways an AI-automated workflow fails, with examples from the HiFi benchmarks, and which layer of the Thunk.AI platform prevents each. Its purpose is practical: when a process fails, the kind of failure decides the fix, and in four cases out of five the fix is not a different model.
1: How an agentic workflow runs
In an agentic workflow, an AI model decides at run time what to do next, instead of following a hard-coded sequence. At each stage the agent runs a loop: it reads its instructions, chooses a tool, calls it, reads the result, and decides what to do next, until the work is done. The tools reach the business's systems: a ticketing system, a document reader, a finance system. This flexibility is what lets an agent handle varied inputs and recover from the unexpected. It is also what makes its behavior non-deterministic, and gives each part of the loop its own way to fail.
2: Instruction errors: the procedure leaves the model to guess
The most common source of unreliable behavior is in the instructions themselves. A standard operating procedure written for trained employees leaves things unsaid, uses terms it does not define, and sometimes contradicts the systems it describes. A model given such a procedure does not know what is missing. It produces a plausible answer, and because the answer is a guess, it can differ from one run to the next.
In the τ²-bench retail work, the store's policy says a request may be transferred to a human agent "if and only if" it cannot be handled within the agent's actions, while the transfer tool's own description says to transfer when the user explicitly asks. The two sources disagree, and no model can know which one the business means. Instructions are ambiguous and incomplete explains why these gaps have to be found and settled at design time, in the Authoring Environment, and enforced in the App Model.
3: Configuration errors: the workflow is built to allow mistakes
A workflow can follow the procedure faithfully and still be configured in a way that lets the agent go wrong. In the first version of the AML alert triage thunk, every step could write every field, so the step that recorded the disposition could override the decision the reasoning step had reached. The rules tool was available in more steps than needed it, so a later step could call it again and record a different answer. Facts were recorded as free text, so values outside the rules, such as a risk rating of "High-ish", could appear.
Configuration errors also run the other way. Early versions of the τ²-bench thunk enforced rules the store's policy did not contain, such as defaulting the payment method for a refund when the policy says the customer must choose one. A workflow that is more constrained than the business intends is as wrong as one that is less.
These are design decisions, and they are caught at design time: in the Authoring Environment, where a design-time AI agent works with the designer and workflow review checks the design before it runs.
4: LLM errors: the model makes a mistake
Some failures are the model's own, even when the instructions are clear and the workflow is well built.
Misapplying exact rules. Procedures contain thresholds, windows, lists and counts. In the AML benchmark, models given the whole procedure as a single step counted two deposits as structuring, treated an 18-month-old report as "over 24 months", and rounded a 79% pass-through up to 80%. The write-up's field-level results show the pattern: the models gathered the facts well and made their mistakes when they had to apply many exact rules at once.
Copying values incorrectly. An agent that carries an id, an amount or a name from one place to another sometimes changes it on the way.
Inventing what is missing. When an output is required and the information is not in the input, a model tends to supply something plausible rather than say it does not know.
Misjudging its own scope. In the IT service management benchmark, GPT-4.1 on its own escalated only 1% of tickets to a person and took incorrect automated actions on 14%. In the AML benchmark, models on their own made between 9 and 40 unnecessary escalations each.
A better model makes fewer of these mistakes, but none of them disappear. The platform reduces them in two ways. The App Model takes away work the model does badly: exact rules move into code tools, facts become typed properties with fixed values, and judgment is confined to narrow questions. The Execution Runtime checks what the model does and feeds failures back for correction.
5: Orchestration errors: the loop lets the agent drift
The loop that runs the agent has its own failure modes. An agent can stop before the work is done and declare itself finished. In the AML benchmark, gpt-5-mini, given the procedure as a single step, finished 17 of 82 alerts without a usable result. An agent can lose track of what it has established, or a later stage can overturn an earlier one. An agent that answers in free text instead of structured actions behaves more variably from one run to the next.
These are prevented by the Execution Runtime, which runs the loop rather than leaving it to the model: the model can respond only with structured tool calls, progress is saved to typed state as it goes, every run must end with an explicit action, and the step's own rules still apply after the agent says it is done. How Thunk.AI runs an AI agent describes these mechanisms.
6: Tool errors: the tools give the agent the wrong footing
Tools are how an agent sees and changes the business's systems, and they fail in their own ways. A tool that returns a whole record when the step needs three fields invites the agent to use the wrong one. A tool with a loose input schema accepts malformed arguments. External systems time out or return errors. A tool with side effects, such as sending a message or filing a report, can repeat the action if a failed step is retried. Real data is also imperfect: the first HiFi benchmark deliberately introduced errors into both its invoices and its vendor reference data, because a process that only works on clean data will not work in production.
Tool errors are prevented in the App Model, by tools with strict schemas that return only what the step needs and that check their own inputs and refuse when a rule is not met, and in the Execution Runtime, which vets every call before it runs, gives each step only the tools it is allowed, and logs every call with its inputs and outputs.
7: Why a stronger model is not the answer
Four of the five kinds of failure have nothing to do with the model's capability. A stronger model does not resolve an ambiguity in the procedure, correct a step that is allowed to overwrite another, stop a loop that ends early, or fix a tool that returns the wrong data. The HiFi benchmarks show the consequence: on Thunk.AI, the gap between the best and worst models shrank from 22.6 points to 0.6 on AML alerts, and from 7.3 points to 1.9 on drug safety cases. What is in a standard operating procedure, and how each part is made reliable describes the design remedies in detail.