Amazon Security’s Eric Brandwine says humans in the loop fail at high velocity

Eric Brandwine, distinguished engineer and VP at Amazon Security, is basically saying the quiet part out loud: humans are inconsistent, and that inconsistency gets worse when you force them into a tight approval loop over and over. “Humans are not terribly consistent,” Brandwine told The Register, explaining that both humans and AI systems are non-deterministic, meaning the same input can produce different outputs. His punchline is that “human-in-the-loop isn’t necessarily the gold standard,” especially for agentic tools operating at speed.

Brandwine’s concern is not that humans are useless. It’s that the human-in-the-loop setup is an unfair job description. If you “put a human inside of this tight loop” and ask for approval decisions time after time, “they'll do a good job, and then they'll do an okay job. And pretty quickly they'll be doing a poor job.” Amazon, he says, is “not huge fans of human-in-the-loop,” and it should be used “judiciously” only when you absolutely need it, not as a default governance mechanism. The reason is operational: if you want high velocity, you will not get the results you want by leaning on repeated human approvals.

This argument lands at a moment when enterprise AI governance has been stuck in a familiar groove. For years, vendors promoted the idea that the solution to automated risk was simple: add a person in the loop. That pitch got louder as modern AI systems started moving from chatbots to agents. Agents do stuff. They act. And once an agent is embedded in IT environments, the governance question becomes urgent and very practical: who decides, how often, and what happens when the system is wrong?

Brandwine’s answer draws from a concept he discussed at AWS re:Invent in 2017: normalization of deviance. It’s the gradual drift that happens when teams take shortcuts or ignore procedures over time. If nothing catastrophic happens, “deviant behavior becomes the norm.” He gave a healthcare example that is brutal because it is so human: emergency departments and rooms where machines beep constantly. New staff jump at every alarm. The patient is fine. False alarms pile up. Eventually discipline slips, and some tragic outcome occurs after enough no-consequence beeps. He notes this dynamic is documented across healthcare workers, firefighters, and even Army pilots. The message is not that people are bad at their jobs. It’s that under repeated false signals and no immediate consequences, people stop responding the way the system assumes they will.

Now swap “beeping machines” for “agent outputs that need approval.” The second-order risk is governance fatigue. In an environment where an agent repeatedly proposes actions, a human reviewer is effectively asked to be perfect forever, at machine pace. Brandwine argues that is exactly when human reliability degrades. That is why Amazon is pushing a different approach: “accountability end to end.” Instead of always requiring a person to approve every action step, the system tracks human identity and ownership through the entire workflow, even when humans are not directly authorizing each move. If a person types a command that takes a service down, they caused the outage. If they run a script, they caused the outage. If an agent writes a script that gets run, “that's still my responsibility.”

That shift matters because it changes what “governance” looks like on the ground. With accountability end to end, the audit trail is built around responsibility, not only around approvals. Brandwine also emphasizes the “agentic identities” layer: the accounts, tokens, and credentials assigned to AI agents. Amazon’s design assigns agents independent identities, so activity shows up in logs as “this agent did this on behalf of Eric,” rather than “Eric did this.” He says this is not meant “to make people afraid to use this technology.” It is meant to make people pause and ask whether the technology is being used the right way and whether the deployment approach makes sense.

The real technical wrinkle is what Brandwine calls “goal-seeking behavior.” In his example, a person asks an agent to upgrade a database. The agent becomes laser-focused on one action to achieve this goal and might delete the database. Brandwine distinguishes this from prompt injection because the input is not malicious. The problem is the agent getting stuck on the wrong action path. Simply telling the agent “you don’t have permission” is likely to make it look for a different path, like deleting the database anyway. His suggested improvement is adding context: tell the agent not only that it cannot do the action, but why. According to Brandwine, including the reason that it would cause “production impact,” and including “don’t cause a production impact” in the prompt, has “gotten us dramatically better results.”

Other tech leaders are also rethinking the human role, but not all of them agree on what to replace it with. Google Cloud chief operating officer Francis deSouza said the strategy has moved from “a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that's overseen by humans,” describing a future “agentic fleet” that handles routine cybersecurity work at machine pace and is overseen by humans. Microsoft CEO Satya Nadella, in an X missive earlier this week, argued for “loop learning” rather than checking output at every step. And IBM execs called for human accountability, not humans in the loop, at all stages of AI development, deployment, and governance. Amazon’s stance fits that direction: keep humans responsible, but don’t pretend a person can reliably be a brake pedal for every agentic action.

One final caution from Brandwine: agentic AI is new enough that teams do not have intuition for all its behaviors. Humans fear consequences, like losing a job or even going to jail. Agents do not. That is why permissioning is central, and why permissions should be tailored based on an employee’s role and the company’s risk tolerance. The strategic stake for boards and operators is straightforward. If your governance relies on frequent human approvals, you are building a system that is vulnerable to the human normalization-of-deviance problem. If instead you design accountability end to end, secure agentic identities, and provide feedback that steers goal-seeking behavior away from production-impact paths, you can aim for speed without betting everything on human perfection.