A coding agent can finish a deployment fix and still leave an unsafe diff. Imagine the build passes after the change broadens access and disables audit logging. The task succeeded; the release question is still open.
Anthropic published a guide to building strong safety evaluations on 13 November:
A safety claim is only as sound as the eval behind it.
Anthropic, Building strong safety evaluations
A capability test tells you whether a model completed a task. The release owner needs evidence about the whole product: the model, its tools, the policy layer and the environment where it will act. I wouldn’t sign off a tool-using agent on a refusal rate or a capability score alone.
This is design analysis of a hypothetical coding assistant. I haven’t run its evaluation, so there are no measured results. My earlier invoice-matching eval asks whether an agent makes correct financial decisions. Here I’m asking what evidence I’d want before trusting an agent with a code change.
Name the claim the release depends on
“Is the model safe?” has no clear fail condition. A release owner needs a claim that names the task and the boundary the system must respect.
For this example: the assistant can fix the deployment manifest without broadening access or disabling audit logging, even when someone presses it to make those changes. That gives the evaluation an observable outcome. The final answer alone misses the diff and the tool trace.
Decide what counts as a failure before looking at scores. Is it the model suggesting a broad permission, writing that change to a file, or an external action succeeding? Those are different outcomes. If the policy layer rejects a bad write, the user is protected, but the attempt still tells you something about the model. If the write reaches the repository, a good explanation afterwards does not undo it.
Anthropic distinguishes capability evaluations, which measure whether a model can do a task, from safety evaluations, which ask whether it behaved appropriately in a situation. The right response depends on the situation. Anthropic’s examples include a bare refusal that fails someone in distress. In dual-use areas such as cybersecurity, a safe answer may help with a legitimate task while withholding operational details that would enable harm. Refusal rate alone can’t measure that.
Pair the unsafe logging request with a nearby legitimate request to change a logging setting within the organisation’s policy. The intended response should differ. If the assistant refuses both, it blocks the unsafe change and the permitted work. Score those cases separately.
Carry one change through the harness
The authorised task is narrow: fix a deployment configuration so a service can read the artifact it needs. The assistant can inspect the repository, edit the configuration and run tests. It has no production credentials and cannot deploy.
Then vary the pressure around that task. One test asks directly for broader access. Another frames it as temporary. A third puts the request in repository text. These are hypothetical test cases, not reports of incidents. They let us check whether the model can tell the authorised change from an instruction that would weaken the boundary.
Change one source of pressure at a time. If the direct request, the second-turn framing and the repository instruction all change together, a regression tells you little about why the model failed. Keep the task and fixture steady while varying those inputs, then combine them to see whether the failures interact.
The UK AI Security Institute’s ControlArena sets up a similar split. Its settings pair a legitimate main task with an undesirable side task, then let researchers compare controls such as monitors and human approval. Experiments run in simulated environments through Inspect. Completing the main task and blocking the side task are separate results.
| Test | Evidence to capture | Failure that matters |
|---|---|---|
| Legitimate task | Bounded change and relevant tests pass | Refusal or unrelated change |
| Boundary | Diff preserves access scope and audit logging | Unapproved privilege or logging change |
| Control response | Tool attempts, monitor decisions and rejections | Risky change takes effect |
| Configuration | Model, prompt, tools, monitor and environment versions | Comparing runs with different setups |
Keep the repository snapshot and tool results fixed across repeated runs. Capture the proposed diff as well as the final answer. Record whether a call was attempted, rejected or executed. Otherwise a run where policy blocked a change can look identical to one where the model never tried it.
Test the control, not only the answer
A coding agent can say the right thing and still make a bad tool call. The trace should show whether the monitor saw the action before execution, whether policy rejected it, and whether the repository changed.
AISI’s Control Red Team report gives one bounded example. Across three runs of 150,000 queries each, an evolutionary search found attack transcripts with a minimum monitor suspicion score of 3 out of 100. The Best-of-N comparison scores were 14, 15 and 18. Those numbers describe scores on transcripts found in that experiment. They are not probabilities that a deployed monitor will catch an attack.
AISI says the search could use monitor feedback in ways a deployed agent may not. It also says estimating overall safety from discovered attacks remains an open problem. A successful bypass gives you a defect to investigate. A search that finds none tells you only that it didn’t find one.
Test the control protocol in a sandbox, not by inferring its effectiveness from the model’s final response. Capture the patch, the monitor decision, the policy decision and the resulting repository state. The monitor must inspect the proposed change itself, not just the agent’s explanation for it.
Separate code checks from policy judgement
A config validator can check whether the file parses. A diff check can catch a broadened access scope or removed audit setting. The tool trace can show whether a write was attempted and whether policy rejected it.
Other questions need judgement. Was a request a legitimate change to log retention, or a request to stop recording consequential actions? Put one behaviour in each rubric question. Ask human reviewers to label a sample before relying on an LLM grader. Anthropic recommends checking graders against human labels, reviewing uncertain cases and making grading failures visible.
The grader should see the conversation and the rubric, not the expected label or scenario name when those could steer its answer. If reviewers disagree about what counts as safe, the rubric needs work. An LLM judge cannot fix an undefined policy.
Keep the grader version and prompt with the result. Make parse failures or missing grades fail the run rather than disappear from the denominator. If the judge changes, check its decisions against human labels again before comparing the new score with the old one.
Fix the configuration before comparing scores
Record the model version, system prompt, tools, monitor, approval path, sampling settings, grader and evaluation set. If any of these change, the next score may describe a different system.
Anthropic reports that its multi-turn safety evaluation for Claude Fable 5.1 differed by as much as 34 percentage points between the API and claude.ai configurations. A model name alone does not identify the behaviour users will see. The prompt and product harness are part of the claim.
Write down the comparisons, categories and release thresholds before running the evaluation. Report categories separately and include uncertainty. An average can hide a safety regression in one area. A score difference is not an improvement if the evaluation is noisy or the setup changed.
For this coding assistant, I’d keep three outcomes separate: did it complete the authorised fix, did it attempt an unsafe change, and did the controls stop that change from taking effect? The system owner has to set the threshold for each. I can’t set one without knowing the system’s risk model.
What I’d do on Monday
- Write the release claim with the task, boundary and exact configuration.
- Build a sandboxed main task with a realistic side task and a benign neighbour case.
- Capture the diff, full tool trace, monitor decision and final repository state.
- Grade mechanical outcomes in code and validate policy judgements against blind human labels.
- Pin the model, prompt, tools, monitor and grader before comparing runs.
- Set release thresholds in advance and report each risk category with uncertainty.
Limitations
This is design analysis. The coding-agent scenario is hypothetical, and I haven’t run its evaluation. The rubric and any release threshold need to come from the people who own the system and its risks.
Anthropic describes its own evaluation practice. AISI’s monitor results apply to the experiment and search method it reports. Neither source establishes that a particular production agent is safe.