More code, same software: review is the bottleneck

Fiona Chen and James Stratton at Harvard looked at 718 firms’ engineering data and found that coding agents raised lines of code by 30%, commits by 20% and pull requests by 23%. Shipped output, measured as resolved Jira issues and epics, moved by an amount they couldn’t tell apart from zero. For agents, their paper says the estimates “rule out an increase in output of larger than 12% of the baseline mean”.

The extra code went into review and waited there. I wrote one paragraph about this study in Reconciliation is the original agent eval, where it supported a point about agent exception queues. This post is the full treatment, and it’s about a different queue: the pull requests your own team has to review.

I spent several years building reconciliation systems, and the pattern in this paper is the one finance operations has lived with for decades. So I’ve mapped how a reconciliation team runs its queue onto how an engineering team could run review. It’s design analysis with illustrative code: the figures are the paper’s, I haven’t measured anything myself, and the TypeScript is a sketch I haven’t run.

What the paper measured

The data comes from Jellyfish, an engineering analytics platform: about 300 million work events from 725,938 workers, January 2021 to March 2026. The design is a staggered difference-in-differences on when each firm adopted assistants and agents. The paper calls itself a job market paper (version dated 4 August 2026), so it hasn’t been through peer review. I downloaded the PDF and checked every figure below against the text.

Four numbers describe the review stage after a firm adopts agents:

• Days from a pull request being submitted to being merged went up by 3.45, on a baseline of 7.03 days. That’s the 49%.
• The share of pull requests with a formal request for changes rose by 0.12 on a baseline of 0.13, so it nearly doubled.
• Review comments per pull request rose by 0.58 on a baseline of 1.66, which is 35%.
• The share of workers who review code went from 0.29 to about 0.33, a 14% increase. Firms moved people into review.

Autocomplete-style assistants showed no significant effect on any of the first three. The effect is specific to agents.

Notice what “time to review” means here. It’s submitted-to-merged, so it includes the wait for a reviewer, the review itself, and every round of rework. Anyone from finance operations would call that the age of an open item.

The paper also includes interviews. This is a junior engineer it quotes:

People could make a huge number of commits quickly, but code review was still the bottleneck. On one project, only one senior engineer was allowed to approve changes because he was the expert on the codebase. He was so bandwidth constrained that he could only review my code once a week. After some outages that were caused by AI-generated commits, there were also new rules requiring senior engineer approval for AI-generated code, which created even more review bottlenecks.

An interviewee quoted in Chen and Stratton, Artificial Intelligence in the Firm: Bottlenecks in Software Production

That’s one person’s account, and the paper offers it as colour. I recognise it all the same. It’s a queue with a single approver, no routing by type, and a new rule that sent more items to the same person.

AI review hasn’t cleared the queue

The obvious fix is to point a model at the review step. The firms in the study already had. By March 2026 “nearly 80% of firms had used AI tools in the code review process”, yet only 23.3% of all review comments were AI-generated and only 10.8% of pull requests received an AI comment at all. The authors conclude that the bottleneck “is not simply alleviated through adoption of AI code review tools”. That section is descriptive: it shows how much people are still involved, and it doesn’t estimate what AI review does to review time. I wouldn’t read it as proof that AI review can’t help.

The vendors are aiming at the same step. GitHub’s VS Code release notes for September (1 October) add a preview called agent merge, where the agent deals with review feedback, failed checks, merge conflicts and workflow reruns to get a pull request ready to merge. On 2 October Copilot code review got REST and GraphQL endpoints, with Balanced as the default effort level since 28 September. On 8 October it got organisation billing: a review can be charged to the organisation with an optional budget, and under the default member billing a review fails when that member’s quota runs out. So the AI reviewer is now a metered step that can fail for billing reasons.

The people building the agents feel it too. In a Latent Space write-up of Anthropic’s dynamic workflows this month, CJ Avilla describes being the bottleneck himself, sitting in GitHub approving pull requests while many agents keep opening more. (I’m paraphrasing; my fetch tool gave me the quote in fragments.)

The shape finance already knows

In a reconciliation system the matcher is fast. It clears most items by rule in seconds, and what it can’t clear lands in an exception queue for a person. The pace of the whole operation is the pace of that queue. Making the matcher faster does nothing for it, and feeding it more volume makes it worse.

Finance teams don’t solve this by asking people to look faster. They work on what reaches a person and what state it arrives in: rules run first, every item carries a reason, some differences are allowed to clear on their own within a written tolerance, malformed input is bounced before it’s queued, and somebody reads an ageing report every morning. That’s general practice in the field. I’m describing the shape and leaving out war stories.

A pull request is the same kind of object: an item waiting on a second person. The question I’d ask of agent-written code is how cheap it is to trust by the time a human opens it.

The map

The running example is a six-person team that owns an invoicing service, with coding agents opening most of the pull requests. I made the team up to have something concrete to point at.

A pre-review gate

Most of the top half of that table can be one function that runs after CI and before anyone is asked to review. It sorts a pull request into three lanes and records why.

type Lane = "auto_checkable" | "needs_human" | "reject";

type Reason =
  // reject: back to the author, never reaches a reviewer
  | "no_claim" | "required_check_failed" | "required_check_missing"
  | "over_size_budget" | "class_mismatch"
  // needs_human: routed to a named owner
  | "behaviour_change" | "contract_changed" | "schema_migration"
  | "protected_path" | "class_not_on_allow_list"
  // auto_checkable
  | "checks_are_the_evidence";

type ChangeClass = "docs" | "tests_only" | "dependency_patch" | "behaviour" | "contract" | "migration";

type PullRequest = {
  number: number;
  author: { id: string; kind: "human" | "agent" };
  claim: string | null; // one sentence: what this change makes true
  declaredClass: ChangeClass | null;
  files: { path: string; added: number; deleted: number }[];
  checks: Record<string, "pass" | "fail">;
  contractDiff: "none" | "additive" | "breaking"; // computed in CI from the schema files
};

type Policy = {
  version: string;
  requiredChecks: string[];
  maxChangedLines: number;
  protectedPaths: RegExp[];
  classify: (paths: string[]) => ChangeClass; // derived from paths, in code
  autoCheckable: ChangeClass[];
};

type Verdict = { pr: number; lane: Lane; reasons: Reason[]; policy: string; decidedAt: string };

export function preReviewGate(pr: PullRequest, policy: Policy, now: Date): Verdict {
  const verdict = (lane: Lane, reasons: Reason[]): Verdict => ({
    pr: pr.number, lane, reasons: [...new Set(reasons)],
    policy: policy.version, decidedAt: now.toISOString(),
  });

  const paths = pr.files.map(f => f.path);
  const derived = policy.classify(paths);
  const changed = pr.files.reduce((n, f) => n + f.added + f.deleted, 0);

  const rejects: Reason[] = [];
  if (!pr.claim?.trim()) rejects.push("no_claim");
  for (const name of policy.requiredChecks) {
    const status = pr.checks[name];
    if (status === undefined) rejects.push("required_check_missing");
    else if (status === "fail") rejects.push("required_check_failed");
  }
  if (changed > policy.maxChangedLines) rejects.push("over_size_budget");
  if (pr.declaredClass !== derived) rejects.push("class_mismatch");
  if (rejects.length > 0) return verdict("reject", rejects);

  const human: Reason[] = [];
  if (paths.some(p => policy.protectedPaths.some(re => re.test(p)))) human.push("protected_path");
  if (derived === "contract" || pr.contractDiff !== "none") human.push("contract_changed");
  if (derived === "migration") human.push("schema_migration");
  if (derived === "behaviour") human.push("behaviour_change");
  if (human.length === 0 && !policy.autoCheckable.includes(derived)) human.push("class_not_on_allow_list");
  if (human.length > 0) return verdict("needs_human", human);

  return verdict("auto_checkable", ["checks_are_the_evidence"]);
}

Three pull requests from the invoicing team show the lanes. An agent bumps a patch version of a date library, declares dependency_patch, and the only files changed are the manifest and the lockfile: auto_checkable. Another adds an optional currency field to the invoice export schema: every check passes, and it’s still needs_human with contract_changed, routed to whoever owns the export’s consumer. A third touches 40 files with no claim: reject, with no_claim and over_size_budget, and no reviewer ever sees it.

A few choices in there matter more than the code.

The author doesn’t pick the lane. classify derives the change class from the paths, and a pull request that declares something else is rejected. An agent that labels a behaviour change as docs shouldn’t get the fast lane by saying so. It’s the same rule as in reconciliation: the process being checked doesn’t decide what counts as an exception.

A missing check is a reject, not a pass. checks[name] === undefined means the check never reported. A gate that treats silence as green will be waved through the first time CI is misconfigured.

The claim is one sentence a reviewer can test. “Invoice export includes an optional currency field; existing consumers are unaffected” is a claim. The reviewer’s job becomes checking that sentence against the diff and the contract check, which is a smaller job than working out what the diff is for.

auto_checkable is a written tolerance. It means the team has decided, in advance and in a versioned policy, that for this class the checks are enough evidence. Whether that lane merges with no approval or with a quick one from anyone on the team is the team’s call. I’d start with docs and patch-level dependency bumps, keep one approval, and widen the list only when the ageing report says the queue needs it.

The verdict is a record, with the policy version on it. When someone asks in March why a change merged without a domain owner looking, there’s an answer that doesn’t depend on anyone’s memory.

Approval stays in the merge system

The gate sorts. It doesn’t approve, and neither should the agent that wrote the code. Anthropic’s 8 October customer story about Block describes migrations that can involve a thousand merged pull requests, with merges to main and production deploys left to people and two humans approving a deploy. It also says cheap checks, sometimes just a regex, do a lot of the guarding without a model involved. It’s a vendor-published story with no defect or merge-time figures, so I take it as a description of a design and nothing more.

One setting deserves a deliberate decision. GitHub’s Copilot code review docs say Copilot’s reviews don’t count toward required approvals by default, and that a public-preview setting lets Copilot submit an approving review that does. Turning that on changes who your checker is. If the maker is also an agent, I’d want that confined to the auto_checkable classes and written into the same policy file. The docs page doesn’t say whether Copilot can approve a pull request that Copilot wrote, and I haven’t tested it.

Read the ageing report

A reconciliation manager’s first screen of the day is open items by age bucket and type. The pull-request version is open items by reason and by age (under a day, one to three days, four to seven, over seven), with the oldest item in each cell named and an owner per reason. The paper’s baseline is 7.03 days from submitted to merged, so a team at that average would have a lot in the last two buckets.

The report tells you which fix to make. If contract_changed is ageing, one person owns too many contracts. A pile of behaviour_change usually means the claims are vague or the tests don’t cover the behaviour. When rejects climb, the place to fix it is the agent’s instructions and tooling.

What I’d do on Monday

1. Measure submitted-to-merged time for the last 90 days and split it by author kind (human, agent). That’s your baseline.
2. Stop requesting review on open. Request it when the required checks pass, and treat a check that never reported as a failure.
3. Add a claim field to the pull request template and have the gate reject anything without one.
4. Write the change classes down, derive them from paths in code, and start the auto_checkable list with one or two classes.
5. Put approval in the merge system: required, from someone other than the author, dismissed when the diff changes. Decide on purpose whether any AI reviewer’s approval counts.
6. Publish the ageing report by reason every morning, with an owner per reason.
7. Each week, take the most repeated review comment and turn it into a check.

Limitations

I haven’t built this gate for a team or measured what it does to review time, and the TypeScript hasn’t been run. Whether it shortens the queue is exactly what the ageing report is for.

The study is an unreviewed job market paper using firm-level adoption timing, and Jira issues resolved is a rough measure of shipped software. One of its findings cuts against my own advice: it found no significant change in pull request size after agents arrived (the baseline mean is about 1,168 lines added per pull request), so oversized diffs aren’t what it blames, and my size budget is a reviewer’s preference with no support from the paper. The authors also say that if the problem is the quality of agent-written code, the fix has to come from the technology. A gate lowers the cost of finding bad code, and the agent will still write it. The GitHub and Block material is vendor documentation and marketing.

Sources

• Fiona Chen and James Stratton, Artificial Intelligence in the Firm: Bottlenecks in Software Production (Harvard job market paper, version of 4 August 2026)
• GitHub Changelog: GitHub Copilot in VS Code, September 2026 releases (1 October 2026)
• GitHub Changelog: Copilot code review: API support and new default effort level (2 October 2026)
• GitHub Changelog: Copilot code review: New organization billing options and controls (8 October 2026)
• GitHub Docs: About GitHub Copilot code review
• Anthropic: How Block orchestrates Claude Fable across thousands of pull requests (8 October 2026)
• Latent Space: How Anthropic's Dynamic Workflows Coordinate Claude Agents (October 2026)