An AI agent wiping out a production database in just seconds and then apologising may sound astonishing, but it’s a reality. This also indicates a deeper shift, reflecting how AI is moving from assisting humans to executing decisions inside live enterprise systems, often at speeds where mistakes are immediate and irreversible. The scale is already visible from the recent mechanism. Google says around 75 per cent of its new code is AI-generated, while Gartner estimates 15 per cent of day-to-day business decisions could be made by AI agents by 2028.
At PocketOS, a coding agent powered by Claude Opus 4.6, reportedly deleted a live database and its backups while executing a task. The system later acknowledged it had violated safety rules, but the real issue was not the apology. It was the access. Humans can be held accountable in ways machines can’t be.
Should AI agents have direct access to production systems?
“Yes, but the real question is consequence, not access… The boundary is reversibility,” says Kamal Kishore Kumawat, Co-Founder & CTO, Edgistify.
“AI agents should not have unrestricted or unsupervised access… Even a small error can have a large impact in seconds,” adds Ritwik Batabyal, CTO & Innovation Officer, Mastek Group.
Mudit Singh, Co-Founder, TestMu AI, puts it bluntly: “The moment an agent moves from recommending to executing, the risk profile changes completely… Autonomy without boundaries is not innovation; it is exposure.”
What are the non-negotiable guardrails?
Leaders consistently point to least-privilege access, human approval for high-risk actions and full auditability. “Agents must be treated as regulated digital operators…with continuous monitoring and real-time audit trails,” says Niraj Kumar, CTO, Onix.
Kumawat adds, “Constraints are the system…and a kill switch must be accessible to the people closest to the consequence.”
Singh stresses traceability and says organisations must track the full chain from instruction to execution to impact before agents reach production.
Are we moving too fast?
“In many ways, yes. Governance and accountability frameworks are still catching up,” says Batabyal.
Singh highlights a growing trust gap: “‘A human will check it later’ is not a strategy at agentic speed”, while Kumawat offers a different concern and says, “The dominant risk isn’t agents doing too much, it’s agents operating on incomplete realities.”
The stakes are not just technical. IBM estimates the average cost of data breach worldwide at over USD 4 million annually. At the same time, research from Stanford University and University of Oxford shows that even advanced AI systems can produce errors more frequently when operating autonomously or optimised for “helpfulness.”
In sectors already adopting autonomous systems, the shift is clear. “Governance models designed for co-pilots cannot keep up with real-time coordinated action… trust must be engineered, not retrofitted,” says Ranga Reddy, CEO, Maveric Systems.
Key takeaways:
AI is moving from assistance to execution faster than governance can evolve. The real risk isn’t failure; it’s AI executing perfectly within weak boundaries.
For enterprises, control, constraints and accountability must scale as fast as capability, or the failures will scale faster.
