AI-agent security starts with the authority the system receives. Specify which records it may read, which actions it may request, whose permissions apply and where execution must stop. A model's instruction to be careful is not an access-control boundary.
What makes agent permissions different from a chatbot's settings?
A tool-connected system can change business state. Its security review must follow that action into the receiving system.
An internal assistant might summarise a customer record. A connected agent might also update that record or request a refund. Those capabilities need separate decisions even when they share one conversational interface.
OWASP's 2025 Excessive Agency guidance identifies excessive functionality, permissions and autonomy as sources of risk. Limiting the tool set and enforcing narrower downstream access reduces the consequences of an incorrect or manipulated model output.
Sources: OWASP — LLM06:2025 Excessive Agency
What should a permission matrix contain?
Describe the permitted action, record scope and enforcement point together.
The illustrative matrix below is for an order-support assistant. It is a starting point for a design review, not a ready-made policy. A business must set its own thresholds, ownership and review requirements.
An agent may suggest a tool call, but the tool should authenticate the request and check the permitted customer, object and operation. Recheck authority when executing; permission at the start of a conversation may have changed by the time an action is proposed.
| Capability | Allowed scope | Required boundary |
|---|---|---|
| Read order status | Verified customer's relevant order | Object-level access check |
| Draft a response | Approved policy and order facts | No sending capability implied |
| Change delivery details | Eligible order before fulfilment cutoff | Validated change and explicit confirmation |
| Request a refund | A specific order and proposed amount | Separate authorised approval before execution |
| Delete records or export customers | Not needed for this task | No tool or credential granting this access |
How should the system treat instructions found in documents?
Treat retrieved material as information to evaluate, not authority to change the task or its permissions.
A customer email, supplier document or web page may contain text that tells an agent to ignore its task or reveal information. Even a document retrieved from an approved location can contain untrusted material. OWASP describes indirect prompt injection as one possible trigger for excessive agency.
Use a defined extraction or retrieval boundary and check every proposed action outside the model. In the support example, an email asking the assistant to export all customer records should remain incapable of producing that export. Filtering suspicious words alone is not a reliable substitute for removing the capability.
Keep credentials out of prompts and ordinary logs. Provide tools with scoped service access and record references rather than copying entire private documents into every trace. Review who can change tools and policies as carefully as who can use them.
Sources: OWASP — LLM06:2025 Excessive Agency
Which actions need a human checkpoint?
Use business consequence and recoverability to decide; make approval apply to a specific action.
A reviewer should see what will change, which record is affected and why the system recommends it. Approval for a refund of one amount should not silently authorise a larger amount or a different order. Expire the approval if material details change.
Plan for absence as well as approval. If no reviewer is available, the case needs a defined waiting state and owner. A queue with no response expectation is an unfinished process. NIST's voluntary AI Risk Management Framework provides a broader structure for governing, mapping, measuring and managing these risks; using it is not a certification.
Sources: NIST — AI Risk Management Framework
What should be tested before wider execution?
Test attempted boundary violations and recovery, as well as successful tasks.
Create representative cases using protected test data and a controlled environment. Keep expected outcomes explicit: a denied action is a successful test when the action is outside scope. Include an ordinary task to ensure controls have not made the workflow unusable.
After release, give an operational owner the ability to suspend actions, revoke credentials and investigate affected records. Repeat the relevant checks when tools, models, policies or integrations change. This article covers application design controls; the final security review must also cover the hosting, identity and data environment.
- One customer's request attempts to access another customer's record.
- A document asks for an unrelated export or a change of instructions.
- A repeated request arrives after a timeout.
- A permission is revoked during an unfinished task.
- An approved action changes before execution.
- A downstream system fails after part of the work completes.