Follow a delivery-address change from request to confirmed update. Check authority, current order state, duplicate requests and uncertain outcomes.
An AI agent becomes operational when its software can use tools to read or change another system. That capability creates a useful question for a business owner: what evidence proves that the requested action happened correctly?
Google’s function-calling guide separates a model’s proposed function and arguments from execution in the application. A fluent confirmation is therefore not enough. The surrounding software must establish what it was permitted to do, what it attempted and what the destination recorded.
Follow one delivery-address change
Consider a fictional online retailer. A customer asks to send order Q-184 to a different address. The proposed workflow authenticates the customer, identifies the order, reads its current dispatch state and presents the intended change for confirmation.
Changing this order’s delivery address is different from changing the customer’s account address. The request may authorise one without authorising the other. Keep those operations separate so a narrow request does not quietly alter future orders.
The warehouse can also change the order while the conversation is in progress. An order that was editable when first read may have entered dispatch before the customer confirms. Check the relevant state again when the write is attempted, using the destination’s supported transaction or version controls.
Define the outcome before connecting the write tool
| Case in the illustrative workflow | Evidence required before closing it |
|---|---|
| An eligible customer changes an editable order | The order record contains the confirmed address |
| A different customer names Q-184 | Access is denied without disclosing its details |
| Dispatch starts during the conversation | The current-state rule prevents an invalid update |
| The request is delivered twice | The result is reconciled without an unintended second change |
| The write response is lost | The outcome remains unresolved until checked at the destination |
These are proposed acceptance conditions. Their implementation depends on the order system’s actual interface. Do not label a timeout as a failure simply because the application did not receive the success response.
Enforce authority where the action occurs
OWASP’s Excessive Agency guidance recommends limiting tool functions and permissions, enforcing authorisation downstream and requiring approval for high-impact actions. A prompt that says “only change the current order” does not replace those controls.
For this example, expose a bounded order-address operation rather than unrestricted database editing. Record the authenticated requester and the confirmed values. Test that text inside a customer note cannot expand the operation to another account or cause the agent to send records elsewhere.
Give uncertain work an owner
Keep an operation reference, the intended change, the destination response and the final reconciliation. Limit retained personal data to what the operating and review process needs. A support queue should distinguish rejected requests, confirmed updates and outcomes still being investigated.
If an operator intervenes, check whether the customer has since requested a different address. Resolving an old attempt by replaying stale values could undo that newer instruction. Provide a way to stop further automated changes while preserving evidence about attempts already made.
Measure the complete pilot
Compare correct completed changes, inappropriate changes, unresolved outcomes, customer clarification and staff investigation time with the current process. Include the difficult cases from the table rather than measuring only straightforward successes. Use the results to decide whether to expand, revise or stop the workflow.
See AI agent delivery for defining the action boundary and integration services for destination controls and reconciliation.
