The short answer
Keep AI agents safe with controls outside the model. Enforce permissions in the database, sign results, cap spending, and keep a halt switch that stops everything. Add approval queues, audit logs, and a practice mode before customers meet the agent. A prompt is an instruction, and it can be ignored or talked around.
Why is the prompt not a security model?
A prompt that says "never issue a refund" is a request. Models usually follow instructions and sometimes they don't, and a customer's message or a document the agent reads can contain text that pushes it somewhere else. You can't test your way to certainty on that, because the input is open-ended.
So we split the design into two questions. What should the agent try to do? That belongs in the prompt. What can the agent possibly do? That belongs in the systems around it, where no text can change the answer. Everything below comes from systems we run: a governed fleet of unattended agents and an SMS booking agent that talks to real customers.
Put permissions in the database
If an agent isn't allowed to do something, the grant shouldn't exist. In our fleet, permissions are enforced by database grants rather than application code. An agent without write access to a table can't write to it, whatever it has been convinced of. There's nothing to talk around.
This also changes how you think about access. Start each agent with almost nothing, then add the specific grants its job requires. Workers in the fleet run in child processes with a minimal environment, so a bad run can't reach credentials it was never handed. Separate keys per agent help too, because you can see who did what and shut off one without touching the rest.
The test is simple. Ask what the worst possible instruction would accomplish if the model followed it completely. If the honest answer is that it could delete customer records or send a mass text, the permissions are too wide, and no amount of careful wording in the prompt fixes that.
Sign and grade what comes back
When an agent reports that it finished, that's a claim. In an unattended system you need to know the result came from the worker you dispatched and wasn't altered on the way back, so every result in our fleet is cryptographically signed.
A signature proves origin, not quality, so a grader checks returned work against a written rubric before it's accepted. That's a second step with a different job, and it catches the most common agent failure, which is confidently reporting success on something half done. A dispatcher that sweeps for work every 15 minutes will happily repeat a mistake all night unless something is checking the output.
Give every failure a ceiling and an off switch
Something will go wrong at 2 a.m., and two controls decide how bad it gets. Spend-capped keys put a ceiling on any bug. A loop that retries forever hits the cap and stops, and you get an annoying morning instead of an alarming invoice. The same thinking applies to anything countable, such as messages sent per hour.
The other control is a halt switch. Ours stops the fleet and kills running workers in one action. Build it before you need it, test it, and make sure a person who isn't an engineer can use it. Our fleet takes commands by SMS and reports in a 6:45 morning briefing and a 4:30 digest, so someone is reading about it daily without having to go looking.
Keep a person in front of risky actions
An approval queue holds an agent's proposed action until a person accepts it. Our SMS booking agent has one for replies that should get a human look, and our prospecting system goes further: its rep assistant proposes actions and never takes them alone. Sorting a mailbox doesn't need this. Spending money, messaging customers, and changing a record of truth do.
Legal rules deserve the same treatment. Consent checks and quiet hours run in code before the model writes a word, so a clever prompt can't skip them. Our SMS system carries a rule set for 20 states, and none of it depends on the model remembering.
Handoff belongs here too. A person should be able to take over any conversation at any point, and the customer should know they are talking to an automated system. Decide how that handoff works before you decide anything about tone, since it is the moment an upset customer will remember.
Log everything and rehearse before going live
An audit log answers the question that comes after every incident: what exactly happened? Record every message and action, with what the agent saw and why it acted. Without that you can't debug a failure, you can't defend a decision, and you can't prove the system is ready for more freedom. Our booking agent logs all of its actions by design.
Practice mode is the last piece, and the one we'd push hardest on. Your team plays the customer and corrects the agent before a real customer ever talks to it. Staff find the awkward cases fast, and they come away trusting the system because they helped shape it.
Together these form a path to autonomy: practice mode, then an approval queue, then limited autonomy for one action type at a time. Each step is earned with measured accuracy on real work. Skip straight to the end and you're relying on the prompt again.
- Practice mode: staff test and correct it, with no customer contact
- Approval queue: the agent proposes and a person sends
- Limited autonomy: proven action types run alone, with the rest still queued
Common questions
Can an AI agent be tricked into doing something harmful?
Yes. Text from customers or documents can steer a model, a problem known as prompt injection. Assume it can happen, and limit the damage by making sure the agent has no permission to do the harmful thing.
Do we need all of these controls for a simple agent?
No. A read-only assistant for internal use needs far less. Controls should scale with what the agent can do, and logging is the one we would keep in every case.
How do we know when an agent can run without approval?
Measure it. Track how often people change its proposed actions, by action type, on real work. When an action type earns your confidence, release that one and keep the rest in the queue.