1. Introduction: The Scalability Paradox of Agentic Systems
In the boardroom, AI agents are promised as the ultimate workers—autonomous, reasoning, and tireless. In the engineering trenches, however, we face a brutal scalability paradox:
the more agents you deploy, the slower, more expensive, and more non-deterministic the system becomes.
When I set out to build the a simulation involving 10,000 independent agents, the traditional approach of calling a Large Language Model (LLM) for every runner’s decision was dead on arrival. To achieve massive scale, you must embrace a counter-intuitive architectural shift:
you have to strategically bypass the LLM.
The goal is to leverage the agentic framework for lifecycle management and telemetry while offloading the heavy lifting to deterministic code. In a production-grade system, your architecture should ensure that adding more runners does not add more tokens.
2. The “Before Model” Callback: The Secret to Deterministic Speed
The technical linchpin of this architecture is the “before model” callback, a feature within the Agent Development Kit (ADK). On the surface, it sounds absurd. Why define an LLM agent only to intercept the call before the model even sees it? As an architect, the answer is “scar tissue.” You want the agent wrapper for its telemetry, its ability to communicate with other agents (A2A), and its structured lifecycle. By using the callback, we “neuter” the probabilistic inference and replace it with millisecond-level deterministic tool calls. This allows us to keep “unit-testable” logic inside a system that is otherwise notoriously gnarly.
The model is required by the LLM agent as in order to create the object but it’s never actually called because before the model call back… intercepts every invocation and returns deterministic tool calls.
By intercepting the invocation, we adhere to the ADK paradigm (maintaining full observability) without paying the “token tax” or the latency penalty of a round-trip to the model.
3. Design-Time vs. Runtime: AI as the Architect, Not the Executor
When being interviewed for engineering roles, candidates often gets asked how they would use AI to solve a specific pathfinding problem. The candidates who suggest using a full LLM to calculate the route at runtime usually don’t get the job. The best answer is to use the AI as the architect during the design phase, not as the executor during the runtime. For the simulation, we faced an NP-hard problem: stitching together a 26.2188-mile path using a specific road network while avoiding walking bridges and indoor paths. Instead of wasting tokens at runtime, we used Gemini in AI Studio during development to research and generate the algorithms. By enabling “grounding with Google Search” and “code execution”, Gemini helped us refine a multi-phase approach:
- Phase 1: A Dijkstra algorithm that utilized haversine weighted edges to stitch together landmarks from a GeoJSON road graph.
- Phase 2: Extending that path with a serpentine waypoint sequence, then trimming the final segment to hit the exact official marathon distance.The AI did the “deep research” and code generation only once. At runtime, the agent simply executes that deterministic Python code. We used the model where it helps (judgment) and skipped it where it doesn’t (repetitive math).
4. Borrowing from Game Development: The “Server-Side Tick” Pattern
Managing 10,000 agents requires a shift toward game development patterns. In a massive multiplayer game, a centralized server uses a “tick” to synchronize state across all entities. We applied this by creating a “Simulator Agent” that acts as the server, orchestrating thousands of “Runner” agents. Within this simulator is a “Tick Agent”. On paper, this is an LLM-defined workflow agent (sequential and looping), but in practice, it is entirely driven by the “before model” callback. Every tick, the agent triggers an “advance tick” tool deterministically. It remains an agent for the purpose of telemetry and state management, but it functions like a high-performance game loop. This ensures that the orchestration of 10,000 runners remains synchronized and token-free.
5. Breaking the State Bottleneck: Why SQL Fails at Scale
When you deploy to a stateless environment like GCP Cloud Run, session management becomes the primary bottleneck. In the “Race Condition” project, we scaled to 50 Cloud Run instances. Because the Global Load Balancer has no affinity for which instance holds a runner’s state, that state must be externalized. We evaluated the standard ADK session stores:
- In-Memory (Local): Useless at scale; state is trapped in a single instance and invisible to the other 49.
- SQL (AlloyDB/Cloud SQL): Too slow. The overhead of SQL queries, transactions, and row-locking was the first thing to break under the pressure of 1,000+ simultaneous sessions.
- Redis Session Service: The winner. We utilized a Redis session service from the Google ADK Community GitHub repo. However, even standard Redis wasn’t enough. We had to subclass the service to perform specific performance tweaks, such as pruning events and preventing blob growth. By keeping the session state streamlined, we could broadcast messages over PubSub and emit updates via Websockets fast enough to visualize 1,000 runners in real-time, even though the backend was handling 10,000.
6. The “Autopilot” Runner: Heuristics Over Inference
The most scalable unit in our system is the “Autopilot” runner. While a standard agent might deliberate over its next move, the Autopilot runner is a specialized extension of the base agent. Instead of an LLM call, it uses heuristics derived from our earlier AI-driven research. It makes decisions about pace, fatigue, and positioning in milliseconds. This transition from probabilistic inference to deterministic code is what allows the system to scale to 10,000 agents without crashing the bank or the server. The AI provides the “judgment” for the initial plan; the “Autopilot” code provides the execution.
7. Conclusion: Engineering the Hybrid Future
The “Race Condition” project proves that the future of AI scale isn’t about bigger models or more tokens, it’s about smarter, hybrid architecture. The philosophy is simple:
Use the model as the architect during design-time, but use deterministic code as the executor at runtime.
As an engineering leader, you must look at your current agentic workflows and identify the “expensive math” that should actually be “free code”. By wrapping deterministic logic in agentic lifecycles, you get the best of both worlds, the observability of an agent and the performance of a compiled algorithm. The future of AI scale is hybrid, and the best architects are the ones who know exactly when to skip the model.
