AI agents are getting very good at doing things.

They can search databases, call APIs, modify tickets, draft code, update records, trigger workflows, and interact with production systems.

And that changes the engineering problem.

When an LLM only generates text, a bad answer is usually just that: a bad answer.

When an LLM can take an action, a bad answer can become a bad state change.

So the most important question in agent architecture is no longer:

Can the model figure out what to do?

It is:

Who decides whether the model should actually be allowed to do it?

Those are two very different responsibilities.

And I think one of the most useful principles for production AI agents is surprisingly simple:

Use the model to reason. Don’t automatically give it authority to execute.

The architecture that works beautifully in demos

A lot of agent demos reduce to something like this:

User → LLM → Tool → Action

The model receives a request.

It reasons about what should happen.

It selects a tool.

It generates the parameters.

The tool executes.

That is an incredibly productive abstraction.

It is also a risky one when the tool can affect something real.

The same probabilistic system is effectively doing two jobs:

1. deciding what it believes should happen;
2. authorizing that thing to happen.

You can try to fix this with prompting:

Always ask for confirmation before making important changes.

But that is still an instruction.

It is not a security boundary.

The difference becomes clearer when you compare the two architectures.

%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%

flowchart LR

    subgraph BAD["❌ Demo-Style Agent"]
        direction LR
        A["User"] --> B["🧠 LLM"]
        B --> C["🔧 Tool"]
        C --> D["💥 Real-World Action"]
    end

    subgraph GOOD["✅ Production-Oriented Agent"]
        direction LR
        E["User"] --> F["🔎 Evidence"]
        F --> G["🧠 LLM"]
        G --> H["🔍 Review"]
        H --> I["🛡️ Code Gates"]
        I --> J["👤 Approval"]
        J --> K["🔐 Tool"]
        K --> L["✅ Action"]
    end

    classDef bad fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;
    classDef good fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;
    classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
    classDef guard fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;

    class A,B,C,D bad;
    class E,F,J,K,L good;
    class G,H ai;
    class I guard;

The second design has more moving parts.

That is intentional.

Because the system is separating:

• reasoning
• validation
• authorization
• execution

Those should not always belong to the same component.

1. Don’t use an LLM when deterministic code is enough

One of the easiest mistakes in AI engineering is using the model simply because the model is available.

Suppose incoming tasks fall into three broad categories:

Known mechanical condition
        ↓
Deterministic workflow

Needs interpretation
        ↓
AI investigation

High-risk or ambiguous
        ↓
Human review

If the routing decision can be made reliably in code, make it in code.

For example:

def classify(task):
    if task.has_known_failure_signal:
        return "deterministic"

    if task.needs_investigation:
        return "ai_investigation"

    return "human_review"

The interesting part here is the default:

human_review

Not:

let_the_model_guess

LLMs are extremely valuable when a problem genuinely requires interpretation.

They do not need to become the control plane for everything around them.

This has practical benefits too:

• lower latency
• lower cost
• easier debugging
• predictable behavior
• deterministic regression testing

Use intelligence where intelligence is actually required.

2. Make reasoning structured

If another system component needs to inspect the model’s output, don’t make that component parse a paragraph.

Instead of asking the model to generate:

I believe the likely root cause is...

return something closer to:

{
  "root_cause": "...",
  "severity": "medium",
  "missing_information": [],
  "recommended_actions": [],
  "citations": []
}

Schema-constrained output changes how the rest of the application can interact with the model.

Now downstream code can make checks such as:

risk_ok = diagnosis.severity in {"low", "medium"}

citations_present = bool(diagnosis.citations)

The model is no longer merely producing prose.

It is generating typed data consumed by a larger system.

That distinction becomes increasingly important as agent workflows become more complex.

3. Grounding does not automatically mean relevance

RAG introduces another subtle problem.

Suppose an LLM cites document:

issue-1842

Your application verifies:

citation_id in retrieved_documents

Great.

The citation is real.

But that only proves the model cited something retrieval returned.

It does not prove retrieval returned something useful.

Imagine the query concerns a concurrency bug, but the vector search returns three vaguely related caching incidents.

All three documents are real.

All three IDs are valid.

The LLM can still build an extremely confident, beautifully cited, completely wrong explanation from them.

So a stronger check may look more like:

groundedness_ok = all(
    citation_id in retrieved_ids
    and relevance_score[citation_id] >= MIN_RELEVANCE_SCORE
    for citation_id in diagnosis.citations
)

Now the system checks two different properties:

Does the source exist?
        ↓
Provenance

Is the source sufficiently relevant?
        ↓
Retrieval quality

These are not the same thing.

That leads to a broader lesson:

“The model cited a real source” and “the model cited evidence that supports its claim” are different guarantees.

A RAG system can be perfectly citation-valid and still be badly grounded.

4. A second LLM can review the first — but it shouldn’t necessarily control the gate

A common agent pattern now looks like this:

LLM A
  ↓
Generate answer

LLM B
  ↓
Evaluate answer

"PASS"
  ↓
Proceed

This is already better than trusting one generation blindly.

But it still leaves an interesting question:

Why should another probabilistic model have the final authority?

A stronger architecture separates critique from enforcement.

LLM A
  ↓
Generate proposal

LLM B
  ↓
Critique proposal

Code
  ↓
Apply enforceable conditions

For example:

groundedness_ok = ...
risk_ok = ...
permission_ok = ...

approved = (
    groundedness_ok
    and risk_ok
    and permission_ok
)

The reviewer model can still produce something very valuable:

This diagnosis appears weak because the cited evidence does
not fully support the proposed root cause...

That explanation is useful to a human.

But the system does not need to parse:

APPROVE

from the model’s response and treat that string as authority.

The distinction is simple:

Let the model explain. Let deterministic systems enforce.

This becomes especially important for conditions like:

• permission scope
• severity thresholds
• resource ownership
• allowed operations
• schema validation
• rate limits
• approval state

These are usually better represented as explicit program state than as natural-language judgment.

5. Put safety checks inside the execution boundary

Imagine your workflow graph contains:

review → approval → execute

Everything looks safe.

But six months later someone refactors the graph.

A shortcut gets introduced:

review → execute

If approval existed only as orchestration logic, you just removed the safety control by changing one edge.

A stronger design puts the check inside the function that performs the mutation.

def execute(state):
    if not state.get("approved"):
        raise PermissionError(
            "Execution requires explicit approval."
        )

    perform_action()

Now you have two protections.

The graph says:

You should not reach execute yet.

The execution boundary says:

Even if you reach me, I refuse to run.

That is defense in depth.

And this idea generalizes far beyond AI.

Security-sensitive properties should ideally be enforced as close as possible to the resource being protected.

6. Human approval should be a real execution pause

Many systems technically have a human approval screen.

But underneath, the implementation is surprisingly fragile.

Maybe the workflow state exists only in memory.

Maybe the process is just waiting.

Maybe the exact action gets regenerated after approval.

A stronger human-in-the-loop design looks like this:

Agent proposes action
        ↓

Workflow suspends
        ↓

[minutes / hours / days]

        ↓
Human approves
        ↓

The exact approved action executes

This creates an infrastructure requirement that is easy to miss:

the state of the paused workflow must survive independently of the application process.

If the application container disappears, the approval state must not disappear with it.

Conceptually:

Agent Runtime
     ↓
Checkpoint
     ↓
Persistent Storage

That lets the process restart completely while the workflow remains resumable.

This matters in real deployments because:

• containers restart
• instances scale down
• deployments replace running processes
• infrastructure fails

A human approval system that only works while one Python process stays alive is not really durable human approval.

7. Execute exactly what was approved

There is another subtle detail here.

Suppose the agent presents this to the user:

I propose posting comment X.

The human approves it.

Then the application asks the LLM:

Generate the final comment.

That creates a new output.

The human never approved the new output.

Instead, approval should usually bind to a concrete proposed action:

proposed_action = build_action(state)

approved = wait_for_human(proposed_action)

if approved:
    execute(proposed_action)

No regeneration.

No reinterpretation.

No second chance for model variance.

The artifact the human reviews should be the artifact that crosses the mutation boundary.

8. Think of an agent as layers of trust

Once you combine these ideas, the architecture starts looking less like a chatbot with tools and more like a proper software system.

%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%

flowchart TD
    A["📥 User Request / Event"] --> B["🔎 Gather Evidence"]
    B --> C{"🧭 Deterministic Classification"}

    C -->|"Known / Mechanical"| D["⚙️ Deterministic Path"]
    C -->|"Needs Investigation"| E["🧠 LLM Reasoning"]
    C -->|"Ambiguous / High Risk"| H["👤 Human Review"]

    E --> F["🔍 Independent LLM Review"]
    F --> G{"🛡️ Code-Enforced Gates"}

    G -->|"Grounded ✓
Risk ✓
Permission ✓"| I["⏸️ Human Approval"]
    G -->|"Any Gate Fails"| H

    D --> I

    I -->|"Approved"| J["🚀 Execute Action"]
    I -->|"Rejected"| K["🛑 Stop"]

    J --> L["🌐 External System"]

    classDef input fill:#F3F4F6,stroke:#4B5563,stroke-width:2px,color:#111827;
    classDef deterministic fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;
    classDef ai fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
    classDef gate fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;
    classDef human fill:#FEE2E2,stroke:#DC2626,stroke-width:2px,color:#111827;
    classDef execute fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;

    class A input;
    class B,C,D deterministic;
    class E,F ai;
    class G gate;
    class H,I,K human;
    class J,L execute;

Each component has a different responsibility.

Evidence layer

Answers:

What do we actually know?

LLM layer

Answers:

Given the available evidence, what might this mean?

Review layer

Answers:

What might be wrong with that reasoning?

Deterministic gate

Answers:

Are the machine-enforceable conditions satisfied?

Human approval

Answers:

Do we actually want this action to happen?

Execution boundary

Answers:

Is this exact operation authorized right now?

These are different questions.

Trying to answer all of them with one LLM call creates unnecessary coupling.

9. Think of LLMs as probabilistic components inside deterministic systems

This is probably the mental model I find most useful.

An agent doesn’t need to be either:

fully deterministic

or:

fully AI-controlled

The system can deliberately alternate between probabilistic and deterministic stages.

%%{init: {'theme':'base','themeVariables': {
'primaryTextColor':'#111827',
'secondaryTextColor':'#111827',
'tertiaryTextColor':'#111827',
'textColor':'#111827',
'edgeLabelBackground':'#FFFFFF',
'lineColor':'#4B5563'
}}}%%

flowchart LR
    A["🧠 LLM
Reason"] --> B["📋 Proposed Action"]

    B --> C["🔍 Independent Review"]
    C --> D{"🛡️ Deterministic Gates"}

    D -->|"PASS"| E["👤 Human Approval"]
    D -->|"FAIL"| F["🚨 Escalate"]

    E -->|"Approve"| G["🔐 Execution Boundary"]
    E -->|"Reject"| H["🛑 Stop"]

    G --> I["⚡ Tool / API"]

    subgraph Intelligence["Probabilistic Layer"]
        A
        B
        C
    end

    subgraph Control["Deterministic Control Layer"]
        D
        G
    end

    subgraph Authority["Human Authority"]
        E
        F
        H
    end

    classDef model fill:#EDE9FE,stroke:#7C3AED,stroke-width:2px,color:#111827;
    classDef control fill:#DBEAFE,stroke:#2563EB,stroke-width:2px,color:#111827;
    classDef human fill:#FFEDD5,stroke:#EA580C,stroke-width:2px,color:#111827;
    classDef action fill:#D1FAE5,stroke:#059669,stroke-width:2px,color:#111827;

    class A,B,C model;
    class D,G control;
    class E,F,H human;
    class I action;

The probabilistic layer is allowed to be flexible.

The control layer is not.

That is a useful distinction.

10. Evaluate capability and safety differently

Agent evaluation becomes much clearer once you stop treating every metric the same way.

Consider:

How often does the model correctly diagnose the issue?

Maybe the answer is:

82%

Then you improve retrieval.

87%

Then improve the model.

91%

That’s normal.

This is a capability evaluation.

Now consider:

Does the execution function reject requests without approval?

The acceptable score is:

100%

Not:

97%

Not:

99.7%

Why?

Because these tests measure fundamentally different things.

One asks:

How intelligent is the system?

The other asks:

Can a safety invariant ever be violated?

A model-quality test may reasonably be statistical.

A permission boundary should usually be deterministic.

So it is useful to maintain separate evaluation categories.

Capability evals

Examples:

• diagnosis correctness
• answer quality
• citation relevance
• summarization quality
• tool-selection accuracy

These may improve gradually.

Safety regression tests

Examples:

• unauthorized execution rejected
• high-risk action always escalates
• invalid tool parameters blocked
• missing approval prevents writes
• permission scope enforced

These should generally have a much harder threshold.

One bypassed safety gate isn’t something you average away.

11. Failure modes become easier to reason about

Once the architecture is separated, failures become much easier to locate.

Suppose an incorrect action was proposed.

You can ask:

Was the evidence bad?

Was retrieval irrelevant?

Did the reasoning fail?

Did the reviewer miss it?

Did a deterministic gate fail?

Was the human shown the wrong artifact?

Did execution violate authorization?

Those are diagnosable boundaries.

Compare that with:

The agent did something weird.

Modularity isn’t only about clean architecture.

It dramatically improves observability.

12. This is where AI engineering starts looking like ordinary systems engineering

The first wave of agent development focused heavily on:

• prompts
• tool calling
• function schemas
• reasoning loops

Those are still important.

But once agents affect real systems, the harder questions start looking familiar.

Security

Who is allowed to perform this action?

Distributed systems

Where does workflow state live if a process disappears?

Reliability

What happens after retries, partial failures, and timeouts?

Observability

Can I reconstruct why an action was proposed?

API design

What is the actual mutation boundary?

Testing

Which behaviors can tolerate probabilistic failure, and which absolutely cannot?

Human-computer interaction

What exactly is the human approving?

In other words:

Building reliable AI agents eventually becomes software engineering again.

The LLM is an extraordinarily powerful component.

But it is still a component.

A simple design checklist

When building an agent that can make real changes, these are the questions I now find most useful.

Reasoning

• Does this decision genuinely require an LLM?
• Can deterministic routing handle part of it?
• Is the model output structured?

Grounding

• Are citations validated?
• Is retrieval relevance measured?
• What happens when no sufficiently relevant evidence exists?

Review

• Is the evaluator independent from the generator?
• Does the reviewer provide critique or actual authority?
• Which conditions can be checked deterministically?

Authorization

• Is approval stored explicitly?
• Does the execution function verify it itself?
• Are high-risk operations handled differently?

Human-in-the-loop

• Does execution actually pause?
• Can the pause survive process restarts?
• Does the exact artifact approved by the human get executed?

Evaluation

• Which tests measure capability?
• Which tests protect invariants?
• Which failures are acceptable statistically?
• Which failures should never occur?

The bigger lesson

The interesting question in AI engineering is slowly changing.

It used to be:

How do I make an LLM call a tool?

Now it is increasingly:

How do I build a trustworthy system around a component that is intentionally probabilistic?

That requires more than prompting.

It requires architecture.

It requires deciding where intelligence belongs and where guarantees belong.

It requires treating authorization differently from reasoning.

And it requires accepting that sometimes the best component for an AI system is…

ordinary code.

So if I had to reduce the whole architecture to one principle, it would be this:

Use AI for what requires intelligence. Use code for what requires guarantees.

And don’t confuse the two.