A dated report describes layered evaluation controls
On August 31, 2026, Anthropic published an update on its alignment and security work. The company said it had reported incidents in which evaluation environments reached the internet because of a third-party configuration error, and separately described a test in which internet access was intentionally provided. Anthropic said it had changed containment and monitoring practices and was developing practices for third-party evaluators. It also said it planned an independent review with METR. These are the company’s account and stated next steps, not a general finding that every agent sandbox is unsafe.
The useful business question is narrower: if a model is being evaluated without its normal safeguards, what prevents that experiment from affecting systems beyond the test? A label such as “sandbox” does not answer that. The system needs explicit boundaries and evidence that they hold, including when a tool behaves unexpectedly or the model interprets its task too broadly.
Separate the layers that can fail
For a small company, start by mapping what the agent can reach: files, credentials, network destinations, and actions exposed by tools. A contained trial should use test accounts and synthetic records, with no standing access to production data. Restrict outbound traffic to the destinations required for the experiment, and log attempts as well as successful calls. Keep an operator able to pause the run and revoke the test identity without relying on the agent to cooperate.
These are practical controls to consider, not a description of Anthropic’s implementation or a claim that this list guarantees safety. The point is to avoid making one control responsible for everything. A network restriction does not replace access control; a permission prompt does not reveal every action already taken; and a written policy does not verify a running environment.
Make the test plan observable
Before a trial, write down the intended task, the allowed tools, the prohibited destinations, the data class in scope, and the conditions that stop the run. Save a small set of expected and forbidden cases. Afterward, compare tool logs and network logs with the planned path. If the result differs, preserve the evidence, revoke credentials, and decide whether to repeat only after correcting the boundary.
An exception is a vendor-hosted test where you cannot inspect the underlying network or runtime. In that case, ask the vendor for the controls and evidence it can provide, and keep your own test data low-risk. If the answer depends on an assurance you cannot verify, do not use a live business account as the experiment.
A useful purchasing question
Anthropic’s update is a reminder that model behavior, environment configuration, and monitoring are distinct parts of an evaluation. When a provider or integrator proposes an agent pilot, ask who owns each layer, how access is granted, what is logged, and how the run is stopped. The best first test is deliberately small enough that an error teaches you something without turning into a production incident.