CrowncodesAdeola

§ Writing13 min read

Reconciliation is the original agent eval

Finance ops learned to trust automated processes long before LLM agents. A map from reconciliation controls to agent-harness equivalents, and common mistakes.

Reconciliation is how finance learned to trust automation it had no reason to trust. Batch jobs, bank feeds, interfaces between systems written in different decades: nobody trusted any of it, so the profession stopped trying to. You compare the process’s output to an independent record, agree what counts as a match, queue whatever doesn’t match, and a named person signs that the books tie out. Then someone else checks that the checking works.

I spent several years building reconciliation systems before I built agents, and I keep watching agent teams reinvent those parts one at a time, usually without the piece that made the original work. A test set, an LLM judge and a dashboard of thumbs-up ratings all help. None of them tells you, on a Tuesday afternoon, whether the thousands of things the agent did this morning were correct. This post maps the reconciliation toolkit onto agent harnesses. It’s design analysis with illustrative code: nothing here is a measured result, and the TypeScript is a sketch I haven’t run.

Update 13 October 2026: I’ve shortened the post and added two things published since: a Harvard study that finds review is what limits teams using coding agents (in the exception queue section), and two papers on how LLM judges go wrong (in the maker-checker section). The argument hasn’t changed.

Two things from late September set this off. On 28 September, Trintech, a close and reconciliation software vendor, launched three finance agents. The release says its Exception Management Agent “groups related exceptions under a shared root cause instead of leaving a team to work through them one line at a time”, and that “every recommendation still comes with its evidence, methodology, approvals and audit trail attached”. That’s a press release for a product I haven’t used, so I’m making no claim about how well it works. What caught my eye is the vocabulary: root causes, approvals and audit trails are what finance buyers already check before they trust anything automated.

Four days earlier François Zaninotto at Marmelab published The State Of AI Harness Engineering 2026, with this line: “Out of the 97 harnesses big enough to test, 60% have neither a test nor an eval.”

My reading of the pair: finance won’t accept an agent without controls, and a lot of the wider agent world still ships without them. I think part of the reason is that most domains don’t have a ledger. When the books don’t balance, an accountant knows. When a support agent gives a subtly wrong answer, nobody finds out.

The map

The running example is a refunds agent for an online shop. It reads a support ticket, looks up the order, decides whether a refund is due and for how much, and proposes it. Refunds above a limit need a human. (It’s deliberately a different agent from the one in my invoice-matching eval post and my permission matrix post. Those cover one case-level eval and what an agent may do. This one is about the controls around an agent once it’s running.)

Reconciliation concept What it does Agent-harness equivalent What teams usually get wrong
Ground truth: the ledger as source of record A record kept independently of the process being checked, which its output is compared against Checking the agent’s claimed actions against the systems it acted on (orders, payments, tickets), not its own trace Grading the agent on its transcript. The trace says “refund issued”; nobody checks the payment provider
Matching rules, strict to loose Ordered rules: exact key and amount first, then fuzzier rules, each tagged with the rule that fired Deterministic graders first (schema, amounts, IDs), then heuristics, an LLM judge last, with the grader recorded per case Starting with an LLM judge for everything, so an exact-match failure and a tone complaint look the same
Tolerance thresholds A written, owned limit for acceptable differences (a rounding penny, an FX band) Explicit pass criteria per field: amount exact, wording may vary, latency under a stated budget Tolerances nobody wrote down, living in someone’s head or in a prompt
Exception queues Every unmatched item becomes a record with an owner, a reason and an age Escalations as typed records with a reason code, evidence and an owner Escalation is a free-text message the agent writes. It can’t be counted, aged or routed
Maker-checker (four-eyes) The person who prepares an entry can’t be the one who approves it The agent proposes; a different principal approves, enforced by the server “Human in the loop” means a confirm button the same session can press, or a second agent that approves the first
Segregation of duties No one role can both initiate and conceal a transaction The agent can’t write to its own logs, edit its own eval set or change its own thresholds The agent’s service account has admin on the tables that record what it did
Audit trail An append-only record of who did what, when, on what evidence Immutable per-action records: actor, inputs, policy decision, result, model and prompt versions Traces kept for 14 days in an observability tool, with no link to the business record they changed
Period close and sign-off At a set point, every item is matched or explained, and a named person signs A regular close over agent activity: exceptions resolved or carried forward with an owner, then signed Nobody owns “was yesterday’s agent activity correct?” Dashboards get looked at, not signed
Root-cause grouping of breaks Group breaks by shared cause so one fix clears fifty lines Cluster failures by failing check and first bad step, then fix the cluster Reading traces one at a time and fixing whichever was most recent
Controls testing Periodically prove each control works, for example by trying to push through a transaction that should be blocked Tests that attack the harness itself: self-approval, over-limit refunds, a missing audit record Testing the agent’s quality and never testing that the guards fire

The source of record lives outside the agent

The most important property of a ledger is that the process being checked didn’t write it. A bank statement comes from the bank. Two records from different places can disagree, and the disagreement is the signal.

Agent evals often lose this. The grader reads the trace, sees a call to issue_refund with a successful response, and marks the case correct. But the trace is the agent’s own account of events. Tool responses can be wrong, retries can double-post, and a partial failure can return success. You’re reconciling a sub-ledger to itself.

For the refunds agent the source of record is the payment provider, so the check compares what the agent claims with what the provider says happened:

type Claim = { runId: string; orderId: string; refundMinor: number; currency: string };
type ProviderRefund = { orderId: string; amountMinor: number; currency: string; refundId: string };

type Break =
  | { kind: "missing_at_provider"; claim: Claim }
  | { kind: "unclaimed_at_provider"; refund: ProviderRefund }
  | { kind: "amount_mismatch"; claim: Claim; refund: ProviderRefund }
  | { kind: "duplicate_at_provider"; orderId: string; refunds: ProviderRefund[] };

export function reconcileRefunds(claims: Claim[], provider: ProviderRefund[]): Break[] {
  const breaks: Break[] = [];
  const byOrder = Map.groupBy(provider, r => r.orderId);

  for (const c of claims) {
    const found = byOrder.get(c.orderId) ?? [];
    if (found.length === 0) breaks.push({ kind: "missing_at_provider", claim: c });
    else if (found.length > 1) breaks.push({ kind: "duplicate_at_provider", orderId: c.orderId, refunds: found });
    else if (found[0].amountMinor !== c.refundMinor || found[0].currency !== c.currency)
      breaks.push({ kind: "amount_mismatch", claim: c, refund: found[0] });
  }

  const claimed = new Set(claims.map(c => c.orderId));
  for (const r of provider) {
    if (!claimed.has(r.orderId)) breaks.push({ kind: "unclaimed_at_provider", refund: r });
  }
  return breaks;
}

This runs over production activity, where a test set can’t reach. Offline evals tell you whether the agent is good on cases you picked. A reconciliation tells you whether what it did today matches reality. Both directions matter. An agent that claims refunds that never happened is broken. Refunds at the provider that no run claims are worse, because something is acting without a record.

Exceptions are records, and the queue sets the pace

In a reconciliation system a break becomes an exception with a type, an owner and a date raised, and it sits in a queue until someone clears it. Agent escalations usually come out as prose (“I wasn’t sure about this one, flagging for review”), which you can’t count, route or age. I’d make escalation a tool with a typed payload and a closed set of reasons:

type EscalationReason =
  | "over_agent_limit" | "policy_ambiguous" | "order_not_found"
  | "amount_disagrees_with_order" | "suspected_injection" | "tool_error";

type Exception = {
  id: string;
  runId: string;
  raisedAt: string;
  reason: EscalationReason;
  subject: { orderId: string; ticketId: string };
  evidence: { kind: "tool_result" | "document"; ref: string }[];
  owner: string | null; // assigned by routing, never by the agent
  status: "open" | "resolved" | "carried_forward";
  resolution?: { by: string; at: string; outcome: "approved" | "rejected" | "amended" };
  versions: { model: string; prompt: string; policy: string };
};

The reason is an enum, so the queue can be routed and reported on. versions means a spike in policy_ambiguous after a prompt change points straight at the change. resolution records what the human decided, which is a labelled case for your eval set (the invoice post calls this the escalation flywheel). The server should also raise exceptions the agent didn’t ask for, such as an amount over limit or a reconciliation break. The process being checked doesn’t get to decide what counts as an exception.

A queue also shows you where the time goes. In reconciliation systems the matcher is rarely the slow part. The pace is set by the pile of items a person has to look at. The best evidence I’ve seen that agents work the same way comes from Fiona Chen and James Stratton at Harvard, who studied coding agents across 718 firms using data from Jellyfish, an engineering analytics platform. Their paper (version dated 4 August 2026) finds agents raised lines of code by 30% and pull requests by 23%, while the effect on shipped output was small and statistically insignificant. They explain the gap like this:

We find evidence of a bottleneck from the code review process: The average time to review a pull request increases by 49%, the share of pull requests with changes requested nearly doubles, and the number of comments per pull request increases by 35%. Firms also reallocate labor toward review activities: the share of workers performing code reviews increases by 14%. This bottleneck persists following the adoption of AI code review tools.

Fiona Chen and James Stratton, Artificial Intelligence in the Firm: Bottlenecks in Software Production

That’s a study of coding agents and code review, and the leap to other agents is mine. But a pull request waiting for review is an exception queue with a nicer UI: make the maker faster and the checker becomes the constraint. If the refunds agent can propose ten times as many refunds, its throughput is whatever the approvers can clear. So measure the queue (volume by reason, age, time to resolve) before you measure the agent.

Maker-checker the agent can’t game

Whoever prepares an entry can’t approve it. Agent systems break this in two common ways. One is the confirm button in the same session: the agent proposes a refund, the UI shows “Approve?”, and the same user (or the same agent in a later turn) presses it. The other is a reviewer agent that reads the first agent’s output and says “looks good”. That adds an opinion but no independent authority.

Two October papers make me trust that opinion less. Ravideshik and Kejriwal (7 October) tested three frontier models as judges of entity alignment, which is record matching: do these two entries refer to the same thing? When the judge could see the label the system had assigned, its ability to tell right from wrong fell to between 0.12 and 0.87 (J-ROC-AUC). With the label hidden it recovered to 0.93 or better. The abstract says label-flip tests confirm the label is the cause, and that stronger judges were more sensitive to it, not less. A day later Kumar, Rathore and Moitra audited six frontier models as judges and found the most consistent one agreed with ground truth only 51% of the time.

I’ve read the abstracts, not the full papers, and neither tests a refunds agent. Still, anyone who has run a maker-checker process will recognise the first result: a checker handed the maker’s answer and asked “is this right?” mostly agrees. So if a judge grades the refunds agent, give it the ticket, the order and the policy, have it reach its own amount, and compare the two in code. The second result is the matching-rules row again. A rule that fires the same way every time can be wrong the same way every time, so a judge needs a labelled sample to be measured against.

Even a good judge isn’t an accountable approver. The control has to be enforced where the money moves: an approval is a token bound to one proposal, and the server refuses it if the approver is the proposer.

type Principal = { id: string; kind: "agent" | "human" };

type Proposal = {
  id: string;
  proposedBy: Principal;
  orderId: string;
  refundMinor: number;
  currency: string;
  digest: string; // hash of the fields above
};

type Approval = { proposalId: string; digest: string; approvedBy: Principal; at: string };

export function checkApproval(p: Proposal, a: Approval, limits: Map<string, number>): string | null {
  if (a.proposalId !== p.id) return "approval is for a different proposal";
  if (a.digest !== p.digest) return "proposal changed after approval";
  if (a.approvedBy.kind !== "human") return "approver must be a human principal";
  if (a.approvedBy.id === p.proposedBy.id) return "proposer cannot approve";
  const limit = limits.get(a.approvedBy.id) ?? 0;
  if (p.refundMinor > limit) return "above approver's limit";
  return null; // ok
}

The digest matters as much as the identity check. Without it, an agent can get approval for a £40 refund and then amend the proposal to £400. If the agent acts “on behalf of” a support rep, the rep is a maker too and shouldn’t be the checker. That’s segregation of duties, and the same principle says the agent’s service account shouldn’t be able to write to the audit table, change its own limits or edit the eval set it’s graded on.

The rows teams skip

Root-cause grouping. When a reconciliation throws hundreds of breaks, nobody works them in order. You sort by the rule that failed and the source file, and the bulk usually come from a handful of causes: a feed arrived late, a reference format changed. Agent teams tend to open the latest failed trace and tweak the prompt, which overfits to whichever trace they read. Give every failure a signature instead (the check that failed, the first bad tool call and its error class, the prompt version), group by it and read three samples from the biggest group. amount_mismatch | get_order | stale_cache at the top of a report is worth more than fifty traces.

Period close. At close, every item is matched or explained, open exceptions are carried forward with a named owner, and someone signs. For an agent that’s a daily or weekly close over its activity. It sounds like bureaucracy until someone asks whether the agent was right last Thursday.

Controls testing. Auditors test a control by trying to push through something it should block. Does the server reject a self-approval, an amended proposal, an over-limit refund, a write to the audit table from the agent’s credentials? These are ordinary integration tests, they belong in CI, and they don’t involve the model at all.

What I’d do on Monday

  1. Name the source of record for every state change the agent can make. If there isn’t one, that’s the first finding.
  2. Write a nightly reconciliation of agent claims against that record, in both directions. Start with counts and amounts.
  3. Turn escalation into a tool with an enum of reasons and a typed payload, and have the server raise its own exceptions too.
  4. Report on the exception queue (volume by reason, age, time to resolve) and give every open item an owner.
  5. Move approval into the server: a token bound to a proposal digest, rejected if approver and proposer match or the amount is over the approver’s limit.
  6. If you use an LLM judge, hide the agent’s decision from it and measure it against a labelled sample.
  7. Write controls tests that try to break each guard, run them in CI, and hold a weekly close that one named person signs.

Limitations

This is design analysis and illustrative code. I haven’t built this refunds agent, the TypeScript hasn’t been run (Map.groupBy also needs a recent runtime), and I have no measured results. The mapping is an analogy: reconciliation works on structured records where “match” can be defined, and for agents that produce text or code it only covers the state changes.

On sources: the Trintech quotes are vendor marketing. The Marmelab figure applies to the open-source harnesses that report examined. The Harvard paper is a job market paper about coding agents, and applying it to exception queues is my inference. The two judge papers are arXiv preprints and I’ve only read their abstracts. None of this is audit, compliance or legal advice.

Sources


  1. 11 min read

    Evaluating an invoice-matching agent before you trust it

    Designing an eval for an agent that matches payments to invoices: cost-weighted outcomes, deterministic graders first, a blinded LLM judge and CI gates.


  2. 12 min read

    More code, same software: review is the bottleneck

    A Harvard study of 718 firms finds coding agents raise code volume but not shipped output, because review became the constraint. What I'd copy from finance ops.


  3. 11 min read

    The outage was in billing

    Two Claude incidents refused requests because of account state, on every model at once. Fallback chains need model, provider and account error classes.