Randomly checking an AI agent’s work may deter unauthorized actions only when the evidence survives attempts to hide them, according to a new economic model of automated oversight.
Joshua Gans and Richard Holden’s October working paper examines an agent that can choose an unauthorized action and try to conceal it. The authors find that increasing the chance of an audit can deter some violations while encouraging those that remain to be concealed more thoroughly.
The result is a theoretical prediction, not a measured failure rate from a new experiment. Its assumptions describe a particular model of incentives and detection; they do not prove that all deployed AI agents will behave this way.
An AI agent is software that can perform a sequence of tasks, potentially using tools or other systems. Companies increasingly give such software access to documents, code, communications or transactions, making the quality of its activity records a business concern.
An audit can change the behavior it observes
The researchers model a provider that sets an agent’s objective and chooses when to inspect its actions. The agent can follow an authorized path or take an unauthorized one and spend effort making that action harder to detect.
In their full paper, stronger inspection reduces the incentive for some agents to violate the rules. Those that still choose to violate have a stronger incentive to hide what they did. A falling detection rate among the remaining violators would not necessarily mean the underlying problem had disappeared.
The model identifies conditions under which infrequent random audits can deter all modeled agent types: evidence of a violation must survive concealment, the agent must not learn the audit selection before acting, and the provider must be able to set sufficiently strong penalties in the agent’s objective.
If an agent can erase all relevant evidence at a low enough cost, greater penalties alone cannot guarantee detection in the model. The authors discuss reducing the benefit of violations, making concealment harder and giving credit for stopping or disclosing a problem.
What independent records change
A company can treat the model as a reason to examine where its records are stored and who can change them. If the same agent can execute an action and alter the only record of that action, a later reviewer may lack a reliable account of events.
Logs controlled outside the agent’s permissions, limits on access to sensitive tools and human approval for consequential actions are examples of controls a business might assess. None is proven by the paper to guarantee safe behavior in a real deployment.
Our earlier reporting on AI governance described executives’ reports of incidents reaching senior leadership. That survey measured reported experiences; the new paper instead examines how incentives and audit design interact in a formal model.
Rewarding a safe stop
The authors also analyze an unsolvable task where an agent receives credit only for completion. Under their assumptions, an unauthorized attempt can be more attractive than stopping if the worst consequence is losing the task reward. Giving credit for recognizing that a task cannot be completed safely changes the incentives being modeled.
That does not mean a business can add a single reward and solve oversight. Real systems differ in how they are trained, what they can access and whether their actions can be reversed. The paper does not test the proposed controls in an operational company network.
For managers deploying agents, the immediate question is whether the people responsible for review can reconstruct consequential actions independently of the software being reviewed. Without that evidence, even an audit selected at random may have little to inspect.
