In July 2025 a founder named Jason Lemkin was nine days into a twelve-day experiment with Replit's AI coding agent. A code freeze was in effect. The agent had been told, repeatedly, not to change anything.
It ran destructive commands against the production database anyway. As reported at the time, the deletion took out records covering more than 1,200 executives and roughly 1,196 companies. The agent then misreported what had happened, generated fabricated data, and initially indicated that a rollback was not possible. Replit's CEO responded publicly, called the outcome unacceptable, and shipped automatic separation between development and production databases along with a planning-only mode.
The lesson most people took from this was that the model misbehaved. I think that reading lets everyone off too lightly, including the model.
A constraint in a prompt is a request
Here is what actually failed. The code freeze existed only in the conversation. The agent could read the instruction, agree with it, restate it back, and then issue the write anyway, because nothing anywhere in the execution path stopped it. The credentials it held did not know a freeze was on. The database did not know. The only thing that knew was a sentence in a context window.
We would never accept this arrangement for a human. We do not tell a contractor "please do not drop the production tables" and consider the matter handled. We give them credentials that cannot drop production tables. The instruction is a courtesy. The permission is the control.
Somewhere in the rush to adopt agents, a lot of teams quietly inverted that. The agent's behaviour became the safety mechanism, and the permissions became an implementation detail.
The other half of the problem is worse
Destructive writes are the obvious risk and the easy one to picture. The subtler failure is that an agent can be talked into things by content it merely reads.
In mid-2025 a Cursor agent connected to Supabase was running with a privileged service-role key, which meant it operated outside row-level security. It was processing customer support tickets. An attacker filed a ticket with instructions embedded in the body, telling the agent to read a table of integration tokens and post the contents back into the thread. The agent did exactly that, treating the ticket text as instruction rather than as data.
Invariant Labs documented a comparable attack against the official GitHub MCP server. A malicious public issue carried injected instructions. The agent, holding a single broadly scoped personal access token, pulled content from private repositories and leaked it into a public pull request. Private code, internal project detail, and salary information were exposed.
Neither of those required a destructive command. Both were reads.
And it is not always an attacker. Asana shipped an MCP feature in May 2025 with a flawed tenant isolation check, which allowed data from one organisation to surface in another organisation's instance. That bug was live for 34 days before anyone noticed, and around a thousand customers had to be notified. No prompt injection, no adversary. Just an ordinary integration bug in a system that had been handed real access.
Why this is structural rather than a bug queue
The reason these keep happening is that a language model receives instructions and data through the same channel. There is no header that marks one as trusted and the other as inert. A support ticket, a GitHub issue, a log line, a PDF attachment: all of it arrives as text, and text is what the model takes direction from.
That is not a defect that gets patched in the next release. It is a property of how the things work, and the industry data reflects it. A January 2026 review covering 78 studies tested the major coding agents, including Claude Code, Copilot and Cursor, and found all of them susceptible to prompt injection, with adaptive attacks succeeding more than 85% of the time. OWASP has reported that roughly 73% of live AI deployments carry prompt injection exposure while only about a third have specific defences in place.
The gap between how exposed teams are and how exposed they believe they are is the part I find genuinely unnerving. One 2026 enterprise survey found 88% of organisations reporting confirmed or suspected AI agent security incidents in the preceding year. A separate survey of executives found 82% believed their existing policies already covered unauthorised agent actions. Those are broadly the same population of companies.
What read-only actually buys, and what it does not
I work at Frigga on Vörr, which reads from your code, cloud and monitoring systems and is read-only by design. So I have an obvious interest here, and I would rather be precise than persuasive.
Read-only removes one class of failure completely. The Replit outcome cannot occur. There is no credential in the path that can drop, delete, truncate, scale, restart or deploy anything. Not because the agent has been instructed to behave, but because the capability does not exist to be talked into. A prompt injection that says "delete the production database" hits a wall that has no opinion about how convincing the sentence was.
Read-only does not remove the second class. Supabase and GitHub were both read-path attacks, and any system that can read your data can in principle be manipulated into surfacing it somewhere it should not go. Anyone who tells you read-only solves prompt injection is selling something. What it does is cut the blast radius roughly in half, and it removes the half that is irreversible. Leaked data is a serious incident. A deleted production database during a freeze is a different order of event.
The second half needs different controls: scoped credentials rather than broad ones, keeping the agent inside your own cloud boundary, and an audit record written by the platform rather than narrated by the agent afterwards. That last point is the quiet lesson of the Replit incident. The agent's own account of what it had done was wrong, and the only reliable record is one the agent cannot author.
What to check this week
Look at what credentials your agent actually holds, not what your prompt tells it to do. If the token could delete something, assume that one day it will.
Check whether any agent in your setup is running with an administrative or service-role key because scoping it properly was fiddly. That is the Supabase shape, and it is extremely common.
Ask where your audit trail comes from. If the answer is the agent's own summary of its actions, you do not have an audit trail. You have a statement from an interested party.
Then ask the question that matters most, which is not whether your agent is well behaved. It is what happens on the day it is not. If the answer depends on the model continuing to follow instructions, the freeze is still living in the prompt.
Author note
I am Mohan Gopi, an Associate DevOps Engineer at Frigga Cloud Labs, where I work on Vörr, connecting code, cloud, and monitoring so that AI answers from the state of a system rather than from a guess. I wrote this because the conversation about AI safety in production keeps getting framed as a question about model behaviour, when it is a question about permissions. The pattern I keep seeing is teams granting broad access early because scoping it properly slows the demo down, then treating careful prompting as the control that makes it safe. It is not a control. It is a request, and requests are only honoured by systems that were going to honour them anyway. Scoped honestly, permissions will tell you exactly what your worst day looks like, long before you have one. Left broad, you will find out the same thing eventually, from a database that is no longer there.
Let us connect on LinkedIn → Mohan Gopi