Written by John Feminella
We now inhabit a world where agents can execute operations in production environments and at the speed of light. The question many are now grappling with: How can we ensure safety for agentic systems with access to powerful operations, without handcuffing what they can accomplish?
The problem
I think about this a lot in the context of the controls we’ve introduced at Two Sigma to limit risks from agents. At a regulated firm, too little safety is an existential risk, and too little innovation is a slow death.
If we want to reap the benefits of agents, we must do so in a way that gives us both safety and freedom to innovate. As John Shedd wrote, “A ship in harbor is safe — but that is not what ships are for.”
For us, we started with five essential ingredients. What’s right for your enterprise will depend on your own exposure to risks and what you want to allow agents to do.
1. Registering agents
To tap the vast potential of our teams to innovate, we need a way for them to share those ideas with each other and for their agents to communicate and work together. An agent registry that allows them to connect to each other is the first, most important step. Without a way for agents (and humans) to understand what’s already been written, we run the risk of wasting a lot of duplicated time, energy, and tokens.
In our case, the registry records each agent’s tools, endpoints, owner, and associated products. One of our goals is to be minimally invasive. We’ve designed our approved templates so that agents built from those templates register themselves, while the bespoke ones are detected heuristically and auto-registered with their owners notified.
2. Risk scoring
Popular agentic harnesses work differently from the way most deployed enterprise agents operate. In general, when you start interacting with a harness like Claude Code, Codex, or Pi, it doesn’t know what kind of request it might get or what purpose it is about to be imbued with — it’s open-ended by design. This presents a challenge for risk evaluation because it could do practically anything.
That’s in contrast to enterprise agents, which almost always have a specific and dedicated purpose. This means that it’s much clearer how we might evaluate and understand these agents. For example, a ticket summarizer differs from an agent reading production market signals.
The registry scores each agent’s risk profile from its declared tools, purpose, and systems, on every update. If an agent is approved for use, the declaration also circumscribes the agent by limiting through policy what it’s allowed to do. The risk scores are based on observed factors like which tools the agent uses, but also qualitative ones like the nature of the agent’s purpose.
Low-risk agents with limited capabilities and access publish without humans involved, and high-risk agents get flagged. This discernment process lets us spend review effort where it matters the most.
3. Egress controls
Agents derive value in part from how they’re allowed to combine and remix information to produce output. But by default, the environments in which our agent processes run don’t trust agents. Instead, agents run behind a proxy that allows only approved destinations. The proxy does not trust agent code, fails closed, and takes the policy for each agent from the registry directly. No one needs to file firewall tickets or maintain brittle configuration.
4. Evaluations
Without a consistent way to do evaluations, teams will have to construct bespoke approaches to understanding model performance against their agent. To us, this would be like asking each team to invent their own unit-testing framework — it’s a wasteful and unnecessary distraction that might be necessary, but ultimately detracts from their ability to innovate.
Instead of bespoke approaches in each team, we aim to have shared evaluations, so that each team writing agents has a consistent way to compare performance. These individual performances against individual models can then be aggregated across teams to understand model performance in the firm’s context. Without this, we’d only have public benchmark numbers, and our experience has taught us that public benchmarks generally don’t have a great correspondence to individual, specific firms.
Our shared framework also handles trace collection and pipeline work, so authors only need to write their usual agent logic and test cases. This means that evals run on every change, just like a CI/CD pipeline.
5. Auto-mode classifiers
If humans need to manually review and approve every agent action, this both slows down the agent and makes the human careless through complacency and alert fatigue. To combat this, we implement a model familiar to folks who use agentic harnesses: “auto mode” classifiers.
If an agent wants to do something that is not expressly forbidden, a two-stage classifier decides what should happen: quick triage that leans toward blocking, then deeper review for unclear cases.
Conclusion
Taken together, these are five of the controls that we’ve put into place for our platform and agentic workloads. This is an evolving space, and what’s right today may not be in six months; we’re constantly engineering our platforms to be ahead of where we need to be to meet internal demand for these capabilities. Our agents will run at machine speed, whether or not their controls do. Successful enterprises build controls that keep up, so nobody has to slow down.