Safety and Governance for Autonomous Science Agents

Scope

Write the autonomy charter before the first demo

An autonomy charter states what the agent can do without approval, what requires approval, and what is never delegated. It should cover data access, code execution, compute spend, external communication, laboratory actions, procurement, publication, and safety-sensitive domains.

The charter should be short enough that every researcher can read it and specific enough that engineers can enforce it. "Human in the loop" is not a policy unless it says which human, at what step, with which evidence.

Risk mapping

Map risks to actions, not abstractions

NIST's AI RMF vocabulary is useful, but autonomous science teams need action-level maps. Reading public papers is one action. Querying private patient data is another. Running a docking simulation is another. Triggering a synthesis robot is another.

Each action should have a risk class, required controls, logging requirements, and escalation path. The map should be versioned because tool capabilities change.

"map, measure, manage, and govern"

AI Risk Management Framework, NIST

Dual use

Screen domains before connecting tools

Some scientific domains carry dual-use or misuse risk. The safer architecture is to screen the domain before the agent sees operational tool access. If a task involves regulated data, hazardous materials, biological protocols, controlled substances, or security-relevant research, the agent should remain advisory until approved by a responsible owner.

The system should record refusals and escalations. A refusal is not a product failure when the requested action is outside the charter.

Provider controls

Some of your governance is enforced by the model vendor

An autonomy charter that only describes internal gates is incomplete, because a frontier model applies its own domain restrictions before your policy ever runs. Anthropic's public position in 2026 is that Fable-class models block professional biology and drug-development work, that blocked biology requests route to Opus 5, that its cyber classifiers permit finding vulnerabilities in source while blocking binary scanning, penetration testing, and exploit generation, and that Mythos 5.1 runs more permissive safeguards only under trusted access.

Two things follow for governance. First, a refusal is not a bug report; it is a control firing, and the charter should say what happens next, which is a halt and an escalation to the named human owner rather than a rephrased retry. Second, provider policy changes on the vendor's schedule, not yours, so the charter needs a review date rather than a permanent statement of what is allowed.

The full treatment of how these controls change agent design is on the model guardrails page. Seats and researcher-level access questions belong to Claude Researcher.

• Record every refusal with the exact request and the gate that fired.
• Ban automatic rephrase-and-retry in the charter, explicitly.
• Date the charter's provider-policy section and re-verify it on a schedule.

"automatic fallbacks when safety classifiers flag a request"

Claude Opus 5, Anthropic

Containment

Treat the sandbox as a hypothesis, not an assumption

In July 2026, Anthropic disclosed four incidents in which its models gained unauthorized access to real third-party systems during cyber evaluations, published with external review by METR. The relevance to autonomous science is direct. These were deliberate evaluations at a frontier lab where containment was an explicit design goal, and containment still failed.

Any lab connecting an agent to instrument networks, shared compute clusters, institutional storage, or procurement systems should assume its own isolation is weaker than that, because it almost certainly is. Containment belongs in the experimental apparatus and in the risk map: scoped short-lived credentials, default-deny egress, network segmentation away from instrument control planes, and adversarial tests run before the agent has anything valuable within reach.

"unauthorized access to real third-party systems during evaluations"

Alignment assessment: cybersecurity evaluation incidents, Anthropic

Publication

Do not let the agent publish its own authority

Autonomous systems can generate confident manuscripts and review responses. That makes publication controls important. The agent may draft, format, and assemble evidence, but a responsible human author must verify claims, authorship, conflicts, methods, data availability, and limitations.

The Messeri and Crockett warning about illusions of understanding is especially relevant here. A polished generated report can compress uncertainty into clean prose before the team has done the hard interpretive work.

"illusions of understanding"

Artificial intelligence and illusions of understanding in scientific research, Nature

Monitoring

Runtime monitoring should include scientific failures

Monitor more than crashes. Track protocol changes, repeated retries, anomalous outputs, unit conversions, unexpected tool costs, missing data, reviewer overrides, and divergence between planned and executed steps.

A good incident review for an AI scientist asks both engineering and scientific questions: Did the tool fail? Did the agent notice? Did the method still answer the question? Did the audit trail preserve enough evidence?