Enterprise agents become useful when they can read context, select tools, perform bounded actions, and recover when the world does not match the plan. That capability comes from architecture, not from a more elaborate system prompt.
The model is one decision component inside a larger control plane. Production design must determine what the agent can see, which actions it may request, what requires approval, how state is stored, and how operators reconstruct a run after something goes wrong.
Start with the authority boundary
Define the maximum consequence of an incorrect action before designing orchestration. Reading a product catalog, drafting a response, creating a reversible ticket, and sending a financial transaction belong to different authority classes.
Use least privilege for tools and data. The agent should receive scoped credentials for the current task rather than inheriting a broad service account. High-consequence actions should require deterministic validation or human approval outside the model.
Separate planning from execution
A useful architecture distinguishes proposed intent from authorized execution. The model may produce a structured plan or tool request. A policy layer validates schema, identity, permissions, limits, and current business rules. The tool adapter then executes and returns a typed result.
This separation makes errors inspectable. It also enables a team to replace a model without rebuilding every integration or embedding policy in prompts that are difficult to audit.
- Context assembly with explicit provenance and access filtering
- Model gateway with versioning, limits, and fallback behavior
- Planner or workflow controller
- Policy and authorization checks
- Typed tool adapters with idempotency
- State, event history, and resumable execution
- Human review and exception queues
- Tracing, evaluation, and operational alerts
Design the failure path first
Agent runs will encounter unavailable tools, contradictory records, malformed responses, rate limits, expired credentials, and tasks that cannot be completed safely. A retry loop is not a recovery strategy. The architecture needs attempt limits, time budgets, compensating actions, and a terminal state an operator can understand.
For long-running work, persist events rather than only the latest conversational state. Event history supports replay, audit, partial recovery, and evaluation of where the decision path diverged.
Evaluate complete trajectories
Response quality is insufficient for an agent. Evaluate whether it selected the right tool, supplied valid arguments, respected authorization, recognized uncertainty, avoided unnecessary actions, and reached a correct final state. Include adversarial instructions inside tool output and retrieved documents.
Separate reasoning from authority
An agent may propose a plan without having authority to execute it. Treat planning, tool selection, authorization, execution, verification, and user communication as separate control points. The model can help with the uncertain parts, while deterministic services enforce identity, permissions, schemas, limits, and transaction rules. This boundary is the foundation of a system that can be tested and audited.
Every tool should have a narrow contract: permitted callers, validated inputs, bounded outputs, timeout behavior, idempotency expectations, and an audit event. Avoid giving the model a general-purpose credential or a single tool that can perform many unrelated operations. Smaller tools make policy clearer and reduce the damage of a mistaken or manipulated action.
Design for interruption and recovery
Long-running work will encounter expired sessions, changed records, unavailable dependencies, and partial completion. Persist the workflow state outside the model conversation. Assign each operation a durable identifier, record what was attempted, and define whether it can be retried, compensated, or must be reviewed. A transcript alone is not a reliable transaction log.
Verification should compare the resulting state with the intended outcome rather than trusting a successful tool response. For consequential operations, the agent should produce an evidence packet containing inputs, selected policy, tool calls, result, and unresolved exceptions. That packet supports human review, debugging, and later evaluation.
Executive decision record
The decision is which judgments may be probabilistic and which actions must remain under deterministic authorization, verification, or human approval. Write that decision before selecting a model, vendor, framework, or implementation pattern. A written boundary keeps technical exploration connected to the operating outcome and makes it possible to explain why the organization advanced, revised, or stopped the work.
Approval should depend on tool contracts, policy tests, durable workflow state, adversarial cases, recovery exercises, and traces that reproduce consequential decisions. The evidence does not need to remove every uncertainty, but it should address the uncertainty capable of changing value, architecture, risk, or ownership. Record the baseline, assumptions, unresolved questions, and the person accepting the next stage.
Failure boundary and operating ownership
The central failure to guard against is allowing a persuasive model response to become authority, transaction state, or proof that an external action completed correctly. Treat that condition as a testable scenario. Define how the system detects it, what users experience, which action is prevented or reversed, and what evidence reaches the person responsible for recovery.
Long-term accountability sits with the product team that owns the workflow, supported by platform, security, and operational owners for shared controls. Supporting specialists can provide platforms, research, review, or delivery capacity, but they cannot substitute for an owner who controls policy and operating change. Name that owner before production and include the ownership path in release evidence and incident procedure.
A practical 90-day application plan
During the first 30 days, convert tool contracts, policy tests, durable workflow state, adversarial cases, recovery exercises, and traces that reproduce consequential decisions into a bounded evidence plan. Assign each artifact to a named contributor, identify the representative inputs required, and agree on the comparison baseline before implementation expands. The objective of this period is to expose the assumption most likely to invalidate the work while the cost of changing direction is still low.
During days 31 through 60, build or instrument the smallest complete workflow that can support the decision about which judgments may be probabilistic and which actions must remain under deterministic authorization, verification, or human approval. Include the real data and authorization path where feasible, record exceptions, and review difficult cases with the people who own the underlying process. Resist adding breadth until the team can explain the measured behavior of this narrow slice.
During days 61 through 90, test the boundary represented by allowing a persuasive model response to become authority, transaction state, or proof that an external action completed correctly. Exercise degraded dependencies, ambiguous inputs, recovery, and handoff rather than demonstrating only successful cases. End the period with a written advance, revise, or stop decision that cites evidence, residual exposure, expected operating cost, and the next authority boundary.
The review should be accepted by the product team that owns the workflow, supported by platform, security, and operational owners for shared controls. That group should confirm not only that the system can work, but that ownership, support capacity, monitoring, and change control are credible. If those conditions are absent, the responsible outcome is another bounded learning stage rather than an unsupported production commitment.
Practical checklist
- Classify tools by consequence, reversibility, and required approval.
- Keep credentials outside the model and scope them per task.
- Validate every tool call against deterministic policy.
- Persist sufficient event history to reconstruct a run.
- Set time, token, action, and cost limits.
- Test prompt injection through users, documents, and tool results.
- Define who receives failed or ambiguous runs.
Engagement scenario
A logistics agent may read shipment events and propose a carrier exception ticket. It can create a draft only when customer, shipment, and reason codes pass deterministic checks. A human approves credits or contractual changes. Every tool request, policy decision, and external response is captured in the run trace.
Continue reading
- [AI agents versus workflow automation](/ai-agents-vs-workflow-automation)
- [Human-in-the-loop agent design](/human-in-the-loop-ai-agent-design)
- [Observability for AI agents](/ai-agent-observability-production)
Sources and further reading
- [Anthropic: Trustworthy agents in practice](https://www.anthropic.com/research/trustworthy-agents)
- [OWASP Top 10 for LLM applications](https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)