AI Reliability in Practice

AI Reliability in Practice

What Is in a Standard Operating Procedure, and How Each Part Is Made Reliable

What Is in a Standard Operating Procedure, and How Each Part Is Made Reliable

Highlights

A standard operating procedure mixes very different kinds of information: the order of the work, the facts to establish, calculations, limits, judgment calls, actions on other systems, reference material, exceptions, and choices it deliberately leaves open. Handing all of it to a model as one block of text treats them alike. They need different treatment.

  • Each kind of information has an approach that makes it reliable. Decompose the work into steps; make the facts explicit and typed; extract deterministic logic into code; represent constraints tightly, where they are enforced; confine judgment to narrow questions; separate deciding from acting; limit what each step can do; ground the agent in reference material; make stopping an explicit outcome; and leave open what the procedure leaves open.

  • Each approach has platform mechanisms. Workflow steps, typed properties and Structure Types, custom code tools, custom AI tools, step conditions and AI checks, scoped tools and plug-in connections, content folders, and human gates.

  • The evidence. In the AML benchmark, the procedure turned directly into seven steps finished only 39 of 82 alerts on gpt-5-mini. The same steps, with these approaches applied, got all 82 dispositions right.

Highlights

A standard operating procedure mixes very different kinds of information: the order of the work, the facts to establish, calculations, limits, judgment calls, actions on other systems, reference material, exceptions, and choices it deliberately leaves open. Handing all of it to a model as one block of text treats them alike. They need different treatment.

  • Each kind of information has an approach that makes it reliable. Decompose the work into steps; make the facts explicit and typed; extract deterministic logic into code; represent constraints tightly, where they are enforced; confine judgment to narrow questions; separate deciding from acting; limit what each step can do; ground the agent in reference material; make stopping an explicit outcome; and leave open what the procedure leaves open.

  • Each approach has platform mechanisms. Workflow steps, typed properties and Structure Types, custom code tools, custom AI tools, step conditions and AI checks, scoped tools and plug-in connections, content folders, and human gates.

  • The evidence. In the AML benchmark, the procedure turned directly into seven steps finished only 39 of 82 alerts on gpt-5-mini. The same steps, with these approaches applied, got all 82 dispositions right.

This article is about design. It describes the kinds of information found in a standard operating procedure (SOP), the broad approaches that make each kind reliable when an AI agent carries out the procedure, and the Thunk.AI platform mechanisms that support each approach. It draws its examples from the HiFi benchmarks, where the same procedures were run with and without these approaches.

1: A procedure is not one kind of instruction

Read any real SOP and it contains several different kinds of statement side by side: "first verify the customer's identity" (an order of work), "record the alert date and the total amount" (facts to establish), "flag structuring if three or more cash deposits below the threshold occur within ten days" (deterministic logic), "only pending orders can be modified" (a constraint), "assess whether the customer's explanation is consistent with their profile" (judgment), "file the report in the case system" (an action), "see the list of high-risk jurisdictions" (reference material), "if the documents are incomplete, refer the case to a senior analyst" (an exception), and, often unstated, choices the procedure leaves to the person doing the work.

A language model given the whole SOP as text treats all of these alike: as instructions to read and interpret. That is appropriate for some of them and a source of error for others. A threshold should be applied, not interpreted. A list of jurisdictions should be looked up, not remembered. The help center puts the principle simply: instructions are read and interpreted, while properties, tools and settings are applied exactly. So each kind of information should go where it can be handled the way it needs to be.

In Thunk.AI, the places it can go are part of the App Model, the platform's representation of a process. Every step in the App Model has four elements: steering (what the step is asked to do), grounding (the information it can see), agency (the tools it may use) and validation (how its results are checked). The work item's state is a set of typed properties, each with its own description, type and constraints. The App Model is one of three layers of the platform; the Authoring Environment helps the designer use it well, and the Execution Runtime enforces it on every run.

2: The kinds of information, and how each is made reliable

Kind of information

Example from a procedure

Approach

Thunk.AI mechanisms

The stages of the work and their order

Intake, gather context, assess, decide, draft, review

Decompose the work into steps, each with one job

Workflow Plan steps and statuses; step run conditions; modules

Facts to establish and record

The alert date, the transactions involved, the risk rating

Make the facts explicit and typed

Typed properties with descriptions and fixed lists of values; Structure Types; Required to finish

Deterministic logic

Thresholds, date windows, counts, totals, eligibility rules

Extract it into code

Custom code tools; tool wrappers

Constraints on values and actions

Only pending orders can be changed; one item change per order; confirm before any change

Represent constraints tightly, where they are enforced

Type constraints; code tools that check and refuse; step finish conditions; conditional tool availability

Judgment

Whether a request is within scope; whether a record explains a red flag

Confine judgment to narrow, explicit questions

Custom AI tools; AI checks; structured evidence that code evaluates

Actions on other systems

Change an order, file a report, close a ticket

Limit what each step can do, and separate deciding from acting

Tools scoped to the step that needs them; read and write tools kept apart; plug-in connections; tool approval

Reference material

Policies, product catalogs, jurisdiction lists

Ground the agent in the source; keep exact lists in code

Content folders; lookup tools

Exceptions and escalation

Incomplete documents go to a senior analyst

Make stopping or asking an explicit, correct outcome

Ask and Error control actions; Required to start and finish; human approval gates

What the procedure leaves open

Which replacement item, which address, which payment method

Do not add rules; offer capabilities

Helper tools that compute options without choosing

The sections below take each approach in turn.

3: Decompose the work into steps, each with one job

A single AI step that carries out a whole procedure has to hold every rule and every fact at once, and has nothing that stops it from finishing early or revisiting a decision. Splitting the work into steps, each with one purpose, narrows what the model has to attend to at any moment. The help center recommends a plan of usually two to ten steps and that each decision belong to exactly one step.

Decomposition alone is not enough. The AML procedure, turned directly into seven steps with its own step text, finished only 39 of the 82 alerts on gpt-5-mini, with 20 safety errors. What changed the result was the structure added to each step, described below. A step is also a boundary: each step should read the facts it needs and write only its own. In the first AML design, any step could write any field, and the disposition step could override the reasoning step's decision.

4: Make the facts explicit and typed

Facts that live in prose are restated, paraphrased and misread by later steps. Facts recorded as typed properties are not. A date is a date, an amount is a number, a risk rating is one of a fixed list of values. The help center notes that a fixed list of values removes misspelled or invented values entirely. In the AML benchmark, typed fields prevented values outside the rules, such as a risk rating of "High-ish" or "RU" for Russia.

Each property also carries a description, which is where the meaning of a value belongs: what counts as the alert date, which amount is meant. Structure Types define records with fixed fields, all required, so an agent cannot drop a field or invent one. Marking a property Required to finish means a step cannot complete without it, which turns "the agent forgot" into a visible, recoverable condition.

5: Extract deterministic logic into code

Anything in a procedure that can be computed from data should be computed by code. This covers arithmetic, dates, thresholds, windows, counts, lookups against exact lists, and rules of the form "if these conditions hold, the answer is this". A model applying such rules in its head makes the errors described in Where unreliable AI behavior comes from: two deposits counted as a pattern, 79% read as 80%.

In Thunk.AI, this logic goes into a custom code tool, written in readable source code and run by the platform. The agent supplies the facts and the tool applies the rules. In the final AML design, a code tool decides which transactions count, applies the thresholds, windows and jurisdiction lists, and returns the disposition, typology, filing deadline and reviewer. The model's job shrinks to getting the facts right, which models do well.

6: Represent constraints tightly, where they are enforced

A constraint stated in the directions ("only modify pending orders") is a request to the model. A constraint enforced by the tool that makes the change is a fact about the process. In the τ²-bench work, every write tool checks the order's status, checks that the order belongs to the authenticated customer, previews the change, and performs it only with the customer's confirmation recorded. The rule that an order's items can be changed only once, after which nothing else on the order can change, is enforced by the tool refusing while another request on the same order is still open.

Constraints belong at the edges: on the property that holds a value, on the tool that performs an action, and on the step's finish condition. Step run and finish conditions are evaluated against the work item's values, not by AI judgment. A finish condition can also include an AI check, a separate model check that the step cannot skip, for example that the outputs match their descriptions.

7: Confine judgment to narrow, explicit questions

Not everything in a procedure can be computed. Interpreting a customer's words, classifying an unstructured document, deciding whether an explanation is plausible: these need a model. The approach is to ask the model a narrow question, in its own context, with only the information it needs, and to get a structured answer.

A custom AI tool does this: it is a self-contained piece of AI work with defined inputs and outputs. In the τ²-bench work, whether a request is within the scope of the agent's actions is decided by an AI tool that sees the customer's words and the list of actions, and nothing else. Where possible, judgment is reduced further to recording evidence. In the AML design, the agent no longer decides whether a bank record explains a red flag. It copies the record's kind, counterparty, transaction types, amounts and dates into a structured record, and code decides what the record explains.

8: Limit what each step can do, and separate deciding from acting

The help center's principle is to minimize capability: give each step only the tools it needs. A step that only looks things up should have no tool that changes anything. In the final AML design, the rules tool is available only in the reasoning step, so the disposition step cannot call it again and record a different answer.

For actions that matter, deciding and acting should be separate. In the τ²-bench design, the AI tool that judges whether a transfer is justified cannot transfer, and the code tool that transfers acts only on that judgment. External systems sit behind plug-in connections with defined interfaces, so a step's access to a system is a declared contract. Individual tools can also require approval before they run.

9: Ground the agent in reference material

Policies, catalogs and lists should be read from their source at the moment they are needed, not recalled from the model's training or from a long prompt. Content folders hold reference documents that steps can search. Exact lists, such as the AML benchmark's jurisdiction lists, belong in code, where membership is a lookup rather than a reading.

10: Make stopping an explicit, correct outcome

Every procedure has cases it cannot settle: missing documents, contradictory data, a request outside its remit. A reliable process treats these as outcomes to produce, not failures to hide. In Thunk.AI, an agent ends every run with an explicit action: conclude, ask a person, or report an error. Even when it concludes, the platform still applies the step's required properties, finish conditions and human gates. People in the loop describes where people should take over.

11: Leave open what the procedure leaves open

Finally, some things in a procedure are deliberately unspecified: which replacement item the customer wants, which of their addresses to use. These should not be turned into rules. In the τ²-bench design they are supported by helper tools that compute options without choosing: a search that returns every item meeting what the customer asked for, and a lookup of the addresses the store already has. Instructions are ambiguous and incomplete explains how open points are told apart from gaps.

12: How the Authoring Environment helps

Choosing where each kind of information goes is design work, and the Authoring Environment is built to help with it. A design-time AI agent plans the workflow with the designer, suggests properties and their types, recommends the smallest set of tools each step needs, and steers away from patterns that make agents brittle, such as one step doing everything. Workflow review then checks the design before it runs: properties without descriptions, tools with loose schemas, steps that write what they have no reason to write, retries of actions with side effects. Problems found at design time are the cheapest to fix, and some of them would otherwise only show up in production.

13: What is left for the directions

After all this, the step's written directions are short. They state the step's objective and its process, explain the judgment the step is asked to make, and point to the tools and properties that carry the rest. The help center suggests splitting any directions longer than about twenty lines into more steps or into tools. Short directions are not a goal in themselves; they are what remains when each kind of information has gone where it can be handled reliably.

This article is about design. It describes the kinds of information found in a standard operating procedure (SOP), the broad approaches that make each kind reliable when an AI agent carries out the procedure, and the Thunk.AI platform mechanisms that support each approach. It draws its examples from the HiFi benchmarks, where the same procedures were run with and without these approaches.

1: A procedure is not one kind of instruction

Read any real SOP and it contains several different kinds of statement side by side: "first verify the customer's identity" (an order of work), "record the alert date and the total amount" (facts to establish), "flag structuring if three or more cash deposits below the threshold occur within ten days" (deterministic logic), "only pending orders can be modified" (a constraint), "assess whether the customer's explanation is consistent with their profile" (judgment), "file the report in the case system" (an action), "see the list of high-risk jurisdictions" (reference material), "if the documents are incomplete, refer the case to a senior analyst" (an exception), and, often unstated, choices the procedure leaves to the person doing the work.

A language model given the whole SOP as text treats all of these alike: as instructions to read and interpret. That is appropriate for some of them and a source of error for others. A threshold should be applied, not interpreted. A list of jurisdictions should be looked up, not remembered. The help center puts the principle simply: instructions are read and interpreted, while properties, tools and settings are applied exactly. So each kind of information should go where it can be handled the way it needs to be.

In Thunk.AI, the places it can go are part of the App Model, the platform's representation of a process. Every step in the App Model has four elements: steering (what the step is asked to do), grounding (the information it can see), agency (the tools it may use) and validation (how its results are checked). The work item's state is a set of typed properties, each with its own description, type and constraints. The App Model is one of three layers of the platform; the Authoring Environment helps the designer use it well, and the Execution Runtime enforces it on every run.

2: The kinds of information, and how each is made reliable

Kind of information

Example from a procedure

Approach

Thunk.AI mechanisms

The stages of the work and their order

Intake, gather context, assess, decide, draft, review

Decompose the work into steps, each with one job

Workflow Plan steps and statuses; step run conditions; modules

Facts to establish and record

The alert date, the transactions involved, the risk rating

Make the facts explicit and typed

Typed properties with descriptions and fixed lists of values; Structure Types; Required to finish

Deterministic logic

Thresholds, date windows, counts, totals, eligibility rules

Extract it into code

Custom code tools; tool wrappers

Constraints on values and actions

Only pending orders can be changed; one item change per order; confirm before any change

Represent constraints tightly, where they are enforced

Type constraints; code tools that check and refuse; step finish conditions; conditional tool availability

Judgment

Whether a request is within scope; whether a record explains a red flag

Confine judgment to narrow, explicit questions

Custom AI tools; AI checks; structured evidence that code evaluates

Actions on other systems

Change an order, file a report, close a ticket

Limit what each step can do, and separate deciding from acting

Tools scoped to the step that needs them; read and write tools kept apart; plug-in connections; tool approval

Reference material

Policies, product catalogs, jurisdiction lists

Ground the agent in the source; keep exact lists in code

Content folders; lookup tools

Exceptions and escalation

Incomplete documents go to a senior analyst

Make stopping or asking an explicit, correct outcome

Ask and Error control actions; Required to start and finish; human approval gates

What the procedure leaves open

Which replacement item, which address, which payment method

Do not add rules; offer capabilities

Helper tools that compute options without choosing

The sections below take each approach in turn.

3: Decompose the work into steps, each with one job

A single AI step that carries out a whole procedure has to hold every rule and every fact at once, and has nothing that stops it from finishing early or revisiting a decision. Splitting the work into steps, each with one purpose, narrows what the model has to attend to at any moment. The help center recommends a plan of usually two to ten steps and that each decision belong to exactly one step.

Decomposition alone is not enough. The AML procedure, turned directly into seven steps with its own step text, finished only 39 of the 82 alerts on gpt-5-mini, with 20 safety errors. What changed the result was the structure added to each step, described below. A step is also a boundary: each step should read the facts it needs and write only its own. In the first AML design, any step could write any field, and the disposition step could override the reasoning step's decision.

4: Make the facts explicit and typed

Facts that live in prose are restated, paraphrased and misread by later steps. Facts recorded as typed properties are not. A date is a date, an amount is a number, a risk rating is one of a fixed list of values. The help center notes that a fixed list of values removes misspelled or invented values entirely. In the AML benchmark, typed fields prevented values outside the rules, such as a risk rating of "High-ish" or "RU" for Russia.

Each property also carries a description, which is where the meaning of a value belongs: what counts as the alert date, which amount is meant. Structure Types define records with fixed fields, all required, so an agent cannot drop a field or invent one. Marking a property Required to finish means a step cannot complete without it, which turns "the agent forgot" into a visible, recoverable condition.

5: Extract deterministic logic into code

Anything in a procedure that can be computed from data should be computed by code. This covers arithmetic, dates, thresholds, windows, counts, lookups against exact lists, and rules of the form "if these conditions hold, the answer is this". A model applying such rules in its head makes the errors described in Where unreliable AI behavior comes from: two deposits counted as a pattern, 79% read as 80%.

In Thunk.AI, this logic goes into a custom code tool, written in readable source code and run by the platform. The agent supplies the facts and the tool applies the rules. In the final AML design, a code tool decides which transactions count, applies the thresholds, windows and jurisdiction lists, and returns the disposition, typology, filing deadline and reviewer. The model's job shrinks to getting the facts right, which models do well.

6: Represent constraints tightly, where they are enforced

A constraint stated in the directions ("only modify pending orders") is a request to the model. A constraint enforced by the tool that makes the change is a fact about the process. In the τ²-bench work, every write tool checks the order's status, checks that the order belongs to the authenticated customer, previews the change, and performs it only with the customer's confirmation recorded. The rule that an order's items can be changed only once, after which nothing else on the order can change, is enforced by the tool refusing while another request on the same order is still open.

Constraints belong at the edges: on the property that holds a value, on the tool that performs an action, and on the step's finish condition. Step run and finish conditions are evaluated against the work item's values, not by AI judgment. A finish condition can also include an AI check, a separate model check that the step cannot skip, for example that the outputs match their descriptions.

7: Confine judgment to narrow, explicit questions

Not everything in a procedure can be computed. Interpreting a customer's words, classifying an unstructured document, deciding whether an explanation is plausible: these need a model. The approach is to ask the model a narrow question, in its own context, with only the information it needs, and to get a structured answer.

A custom AI tool does this: it is a self-contained piece of AI work with defined inputs and outputs. In the τ²-bench work, whether a request is within the scope of the agent's actions is decided by an AI tool that sees the customer's words and the list of actions, and nothing else. Where possible, judgment is reduced further to recording evidence. In the AML design, the agent no longer decides whether a bank record explains a red flag. It copies the record's kind, counterparty, transaction types, amounts and dates into a structured record, and code decides what the record explains.

8: Limit what each step can do, and separate deciding from acting

The help center's principle is to minimize capability: give each step only the tools it needs. A step that only looks things up should have no tool that changes anything. In the final AML design, the rules tool is available only in the reasoning step, so the disposition step cannot call it again and record a different answer.

For actions that matter, deciding and acting should be separate. In the τ²-bench design, the AI tool that judges whether a transfer is justified cannot transfer, and the code tool that transfers acts only on that judgment. External systems sit behind plug-in connections with defined interfaces, so a step's access to a system is a declared contract. Individual tools can also require approval before they run.

9: Ground the agent in reference material

Policies, catalogs and lists should be read from their source at the moment they are needed, not recalled from the model's training or from a long prompt. Content folders hold reference documents that steps can search. Exact lists, such as the AML benchmark's jurisdiction lists, belong in code, where membership is a lookup rather than a reading.

10: Make stopping an explicit, correct outcome

Every procedure has cases it cannot settle: missing documents, contradictory data, a request outside its remit. A reliable process treats these as outcomes to produce, not failures to hide. In Thunk.AI, an agent ends every run with an explicit action: conclude, ask a person, or report an error. Even when it concludes, the platform still applies the step's required properties, finish conditions and human gates. People in the loop describes where people should take over.

11: Leave open what the procedure leaves open

Finally, some things in a procedure are deliberately unspecified: which replacement item the customer wants, which of their addresses to use. These should not be turned into rules. In the τ²-bench design they are supported by helper tools that compute options without choosing: a search that returns every item meeting what the customer asked for, and a lookup of the addresses the store already has. Instructions are ambiguous and incomplete explains how open points are told apart from gaps.

12: How the Authoring Environment helps

Choosing where each kind of information goes is design work, and the Authoring Environment is built to help with it. A design-time AI agent plans the workflow with the designer, suggests properties and their types, recommends the smallest set of tools each step needs, and steers away from patterns that make agents brittle, such as one step doing everything. Workflow review then checks the design before it runs: properties without descriptions, tools with loose schemas, steps that write what they have no reason to write, retries of actions with side effects. Problems found at design time are the cheapest to fix, and some of them would otherwise only show up in production.

13: What is left for the directions

After all this, the step's written directions are short. They state the step's objective and its process, explain the judgment the step is asked to make, and point to the tools and properties that carry the rest. The help center suggests splitting any directions longer than about twenty lines into more steps or into tools. Short directions are not a goal in themselves; they are what remains when each kind of information has gone where it can be handled reliably.