On 19 November, OpenAI described a Performance Factory that uses Codex agents to investigate ChatGPT performance regressions. Its Regression Workflow starts a task when an alert fires. An agent investigates, records its findings and proposes a fix for an engineer to review; after deployment, another agent checks whether performance returned to baseline (OpenAI’s account).
OpenAI reports that the workflow tracked more than 1,700 investigations and root-caused over 600 cases in just over three months. Those are useful counts of work entering the system and reaching a diagnosis. They don’t tell us how many proposed fixes were deployed, how many held, or whether customers saw fewer slow requests.
What I would borrow is the loop, without treating it as proof of results. A performance task should stay open until a fixed, user-facing measurement says the change helped. This is design analysis with illustrative TypeScript: I haven’t run the code or measured an agent workflow. The example is a hypothetical service, not an incident from my work.
The last step is the hard one
Diagnosis and verification are different jobs. A trace can make one database call look suspicious; a patch can remove it; neither fact proves that the service is faster for the people using it. If the same agent writes the proposed fix and decides what counts as success, the task can close on a convincing explanation instead of evidence.
OpenAI’s description separates the investigation from the post-deployment check by using another agent. That separation is useful operationally, but a second run is not automatically an independent evaluation. I would keep the metric definition, comparison window and close rule in application code or configuration that the model cannot rewrite as part of its task.
This reminds me of reconciliation pipelines. Moving an item to “matched” is not enough if the record still disagrees with its source of record, so the state change needs a check against evidence outside the matching step. A performance fix needs the same kind of boundary.
The running example: a slow invoice endpoint
Suppose a release makes an invoice-ingestion endpoint slower. The monitor raises an alert for the endpoint’s p95 latency. An operational workflow opens a durable task and records the service, route, candidate version, alert query, metric-policy version and the time window that triggered it.
The investigating agent can read traces and the relevant code, then propose a small patch. It can explain why the change might reduce latency and show the tests it ran. A person reviews that patch before anyone uses it. None of this gives the agent authority to change the success metric or to close the task.
After deployment, the verification step reads telemetry from the candidate and a control version. The evaluator asks a narrower question than “does this look fixed?”: did the candidate improve the chosen SLI under a comparable sample, without worsening the error rate or breaking the service’s absolute SLO? The answer comes from measurements and a policy version, not the agent’s summary.
| Stage | Record to keep | What stops the task closing |
|---|---|---|
| Alert | Route, candidate version, query and policy version | No clear owner or no reproducible signal |
| Investigation | Evidence links, diagnosis and proposed diff | A plausible cause is not a verified fix |
| Review and rollout | Reviewed change, tests and rollout version | The patch has not reached the measured candidate |
| Verification | Candidate/control window, sample and SLI result | Missing, incomparable or failing evidence |
The task needs more than a binary open-or-closed state after deployment. “Improved” means the policy’s criteria passed. “Unchanged” means the proposed fix didn’t move the selected metric enough. “Regressed” means the candidate crossed a failure threshold. “Insufficient data” means there isn’t a trustworthy comparison yet. Only the first state should make a task eligible for closure, and an absolute SLO breach still blocks closure.
Compare a candidate with a control
A before-and-after graph looks persuasive and is easy to misread. Traffic mix changes through the day; a dependency may recover on its own; load can rise or fall while the agent is making its change. If the metric improved after deployment, that sequence alone doesn’t establish that the patch caused the improvement.
Google’s SRE Workbook defines a canary as a partial, time-limited deployment evaluated against a control. It recommends metrics tied to service health and comparisons that attribute a change to the candidate. It also warns that before-and-after comparisons are vulnerable to time-related changes, and that the candidate needs enough representative traffic for the result to mean anything (Canarying Releases).
That changes the evidence I would ask the verification step to collect. The candidate and control need the same route, request class and measurement window. The selected latency metric should be tied to a user-facing service objective; a general CPU reading can move without telling us whether requests got worse. The comparison should include the error rate as well as latency, because a faster endpoint that rejects more requests has not met the intent of the fix.
The policy also needs an absolute limit. A candidate might look better than a degraded control while both are still outside the service objective. Relative improvement is not enough. The SRE workbook makes the same distinction between evaluating the canary against its control and checking the service against its SLO.
A canary that sees too little traffic should not produce a green verdict. Performance problems often show up under load, and a short low-volume window can miss them. If the service cannot produce a representative comparison, the task stays open for more evidence or a different test. That is an operational state, and the model shouldn’t fill the gap with confidence.
Put the stop condition in code
The model can explain the evidence. A deterministic evaluator should decide whether the work item is eligible to close. This sketch assumes the query has already selected comparable route and request classes. The service owner configures the policy values; none are measured here.
type Snapshot = {
version: string;
route: string;
windowId: string;
requestClass: string;
samples: number;
p95Ms: number;
errorRate: number;
};
type ClosePolicy = {
version: string;
minSamples: number;
requiredLatencyGain: number;
maxErrorRateIncrease: number;
p95CeilingMs: number;
errorRateCeiling: number;
};
type Decision =
| { state: "eligible_for_close"; policy: string }
| { state: "keep_open"; reason: string; policy: string };
function verify(candidate: Snapshot, control: Snapshot, policy: ClosePolicy): Decision {
if (candidate.route !== control.route ||
candidate.windowId !== control.windowId ||
candidate.requestClass !== control.requestClass ||
candidate.samples < policy.minSamples ||
control.samples < policy.minSamples ||
!Number.isFinite(candidate.samples) ||
!Number.isFinite(control.samples) ||
!Number.isFinite(candidate.p95Ms) ||
!Number.isFinite(control.p95Ms) ||
control.p95Ms <= 0) {
return { state: "keep_open", reason: "incomparable_or_insufficient_data", policy: policy.version };
}
const latencyGain = (control.p95Ms - candidate.p95Ms) / control.p95Ms;
const errorIncrease = candidate.errorRate - control.errorRate;
const withinAbsoluteLimits = candidate.p95Ms <= policy.p95CeilingMs &&
candidate.errorRate <= policy.errorRateCeiling;
if (withinAbsoluteLimits && latencyGain >= policy.requiredLatencyGain &&
errorIncrease <= policy.maxErrorRateIncrease) {
return { state: "eligible_for_close", policy: policy.version };
}
return { state: "keep_open", reason: "policy_not_met", policy: policy.version };
}
The query adapter also needs to reject missing or invalid readings and establish that the sample is representative; a count alone cannot do that. The task service should store the input snapshots, the policy version and the returned decision. It should not accept a model-authored field such as fixed: true as its close command. A person can still inspect the evidence or decide that the alert was invalid, but that is a separate, recorded resolution with an owner and reason.
This is also where a second agent can help without being the judge. It can retrieve the measurement, explain the comparison and call out a noisy sample. The task service owns the policy result. If the agent and evaluator disagree, the task stays open until someone resolves the discrepancy.
What I’d do on Monday
- Write the service’s user-facing SLI and absolute SLO into versioned configuration before asking an agent to investigate.
- Make every alert create a task with the affected route, candidate version, query and owner.
- Give the investigating agent read access to relevant telemetry and code, then require a reviewed patch before deployment.
- Compare the deployed candidate with a control in the same measurement window; record request class and sample size.
- Treat missing data, a mismatched window or a low-volume sample as
keep_open, never as a pass. - Store the measurements and policy version with every close decision so the next person can reproduce it.
Limitations
This is design analysis, and the TypeScript is illustrative and unrun. I have not built this workflow or measured whether it reduces incident time, improves latency, or saves engineering effort.
OpenAI’s post is a vendor description, and its investigation counts are not outcome measures. Direct retrieval of the article was blocked in this run; I used the text returned for its official page in search results and have limited the claims to that description. The post does not establish the workflow’s metric definitions, canary design, thresholds or rollback policy.