Orchestration and Agentic Topologies

How to read this chapter

This chapter follows the layer around the model and the mechanisms that keep its cost out of a token forecast.

1. Follow one task through the layer — one answer traced across the systems that produced it.
2. The memory layer — read-time savings, write-time costs and storage growth.
3. Tool servers and MCP — what the July 2026 protocol revision changes for capacity.
4. The harnesses — the questions that decide which line items an agent framework creates.
5. Topologies — context replay, fan-out, and the case for a single thread.
6. Concurrency, retries and held resources — Little’s law, multiplying retries, and resuming instead of restarting.
7. The operating inventory — what to keep visible for each component.
8. What the review inherits — attribution, failed tasks and the value of completed work.

Follow one task through the layer

Follow that hypothetical support task one layer deeper. Before calling the model, the assistant embeds the customer’s question and searches its memory store, which costs embedding tokens and a vector query. The draft suggests a refund, so the assistant checks the order system through a tool server with its own latency, limits and perhaps charges. After the reply goes out, a memory update may make further model calls to decide what to keep. A trace links these operations to the task and has its own ingestion and storage bill. The exact path depends on the application; each optional step needs a purpose.

The task is one outcome with consumption across several systems, owned by different teams and billed on different invoices. How much of its cost falls outside the model bill depends on the design, so measure the share. Chapter 8 logged a task identifier with every model call; attribution by agent, workflow and customer needs it on every event, plus a policy for shared services such as a memory cluster. OpenTelemetry context propagation carries the identifier between services, and the current Model Context Protocol revision carries it into tool calls.¹²

A retry repeats events across every system in the path. A memory write that fails silently becomes a missing fact that the agent later retrieves incompletely, producing a worse draft that takes another attempt. Some failures in this layer appear as quality drift rather than outages, making an orchestration fault look like a model problem.

The memory layer

A memory layer trades work at write time for context at read time. On one long-conversation benchmark, Mem0’s paper reports about 1,800 tokens of retrieved memory against 26,000 for the full history, a 95th-percentile response time of 1.4 seconds against 17, and somewhat lower accuracy.³ Memory vendors dispute one another’s figures, so I would carry the direction of that trade into a forecast, not the numbers. Writing has its own meter. Graphiti, the open-source framework at the core of Zep’s service, runs a language model over each new episode to extract entities and relationships, and by default limits ingestion to ten concurrent operations to avoid the provider’s rate-limit errors.⁴ Ingestion competes with the agent for any limits they share, unless it runs in the background, off the response path, during a quieter interval.⁵

Storage growth depends on what each design keeps and what, if anything, it removes. Mem0’s update step can delete a contradicted fact, though its memory decay feature changes only search ranking.⁶⁷ Graphiti marks superseded facts invalid instead of deleting them.⁸ Letta, which grew out of the MemGPT research, now keeps memory as files in a git repository, a shift it announced in March 2026; files at the memory root load into the system prompt on every turn, and every edit is committed.⁹¹⁰ Preserved history may be exactly what the application needs, but storage then grows with every change, and a deletion request has to reach the history.

Vector storage shows which choices matter. Qdrant’s capacity guide sizes dense vectors as points × replicas × dimensions × bytes per dimension, adds a term for the HNSW search graph, and keeps both in RAM by default.¹¹ For illustration, one million new memories a month at 1,536 dimensions, stored as 32-bit floats with two replicas, need 12,000,000 × 2 × 1,536 × 4 bytes ≈ 147 GB a year; the graph, at the default of 16 links per point, adds 24,000,000 × 16 × 2 × 4 bytes × 1.2 ≈ 3.7 GB. That is about 151 GB of RAM per retained year before payloads and the headroom Qdrant recommends, and three years of retention triples it. Fewer dimensions, or a quantized copy in RAM with the originals on disk, shrink the vector term if search quality survives the test. Before comparing prices, agree with the workload owner and privacy specialists on what the store may keep and how deletion reaches indexes, derived memories, history and backups; a forecast that assumes indefinite retention should say so.

Tool servers and MCP

The Model Context Protocol connects AI applications to tools and data. Its current revision, dated July 28, 2026, made the protocol stateless: the initialization handshake and protocol-level sessions are gone, and a server that needs state across calls issues an explicit handle and keeps the state itself.¹² A stdio server is a subprocess the client launches, so its process count follows how clients are deployed and how long they run rather than the number of users, and it can still call remote APIs with real credentials and charges. A Streamable HTTP server runs independently and can serve many clients; without protocol sessions it can sit behind an ordinary load balancer, with its state in whatever store backs the handles. Inventory the server, the services it calls and that store.

Two other changes reach the bill. Servers should now list tools in a deterministic order, because tool definitions often sit in the model’s context and a stable order improves prompt-cache hit rates; a server that reshuffles its list can raise the input cost of every agent using it. And a broken response stream loses the in-flight request, which the client must send again, so a tool that is not idempotent may act twice. Sequential tool calls add latency, and parallel ones create concurrency at the tool and the systems behind it. A third-party server also needs the dependency review any external service receives, of its operator, data access, limits and continuity; a listing in the official registry, still in preview, establishes none of them.¹³

The harnesses

An agent harness is the runtime that holds the loop: the system prompt, the context window, the tools, the memory interface and the execution environment. Its design decides which line items exist at all. Four questions cover most of the cost:

1. What does it send on every turn? Its instructions and tool definitions ride along on each call. Pi advertises its minimal system prompt as a token saving; Claude Code warns when the descriptions of custom subagents pass 15,000 tokens.¹⁴¹⁵
2. Can it fan out, and onto which model? Pi ships without subagents by design. A Claude Code subagent has its own context window and tools, can run on a cheaper model, and counts toward the same usage limits as the main conversation.¹⁶
3. Where does code run? Hermes Agent can run commands locally, in Docker, over SSH or in hosted sandboxes, some of which hibernate when idle, so one harness can mean a laptop or a metered container fleet.¹⁷
4. Does a failed run resume or restart? Anthropic’s managed agent service keeps an append-only session log outside the harness, so a replacement harness continues from the last event; Pi’s experimental Durable package, released October 1, 2026, checkpoints every step.¹⁸¹⁹

A managed runtime adds a fifth question: what it leaves in your inventory. Microsoft Foundry Agent Service’s basic setup keeps agent data in Microsoft-managed storage; its standard setup uses the customer’s own Azure Storage, Azure Cosmos DB and Azure AI Search resources for files, conversation state and vector stores, and needs at least 3,000 RU/s of Cosmos DB throughput per project. Provisioned throughput is billed by the hour whether or not an agent runs, while serverless mode bills for consumption in a single region.²⁰ For our support assistant, I would connect retained conversations and retrieval activity to those resources and assign retention, throughput and cost ownership with the platform team. A managed runtime can reduce operating work without making the surrounding data services disappear.

Provider-held state also has a lifecycle, so whoever hosts the runtime, I would keep a copy of any conversation history the business needs; when OpenAI retired its Assistants API on August 26, 2026, developers had to rebuild earlier conversations from messages they had stored themselves.²¹

Topologies

Start with that single agent, because its cost is the easiest to underestimate. Each turn feeds the model the system prompt, the tool definitions and everything that has happened so far. With a prefix of $P$ tokens, $\Delta$ tokens added per turn and $n$ turns, the task consumes

$$nP + \Delta\,\frac{n(n-1)}{2}$$

input tokens. With invented values of $P = 10{,}000$ and $\Delta = 2{,}000$, a 20-turn task consumes 200,000 + 380,000 = 580,000 input tokens and a 40-turn task 400,000 + 1,560,000 = 1,960,000. Twice the turns cost 3.4 times the input, so a forecast built on the average number of turns understates the average task.

Prompt caching changes both the price and the headroom of that loop. At Anthropic’s standard rates, a five-minute cache write costs 1.25 times the base input price, a read costs 0.1 times, and each read renews the lifetime.²² If each turn reads the history so far and writes only its new tokens, the 40-turn task reads 1,872,000 tokens and writes 88,000, for 1,872,000 × 0.1 + 88,000 × 1.25 = 297,200 base-rate tokens, about 15 percent of the uncached price. On most Claude models, cached reads also do not count toward the input-tokens-per-minute limit, so caching stretches the quota as well as the budget.²³ A timestamp near the top of the system prompt is enough to break the identical prefix the discount depends on.

Anthropic reported that in its data, agents used about four times the tokens of chat interactions and its multi-agent research system about fifteen times; early versions spawned 50 subagents for simple queries until the prompts carried explicit scaling rules.²⁴ That is one system’s measurement, not an upper bound or a multiplier for every forecast.

For illustration, suppose four workers each receive a 10,000-token brief and produce 5,000 output tokens. Their calls consume 4 × 10,000 = 40,000 input tokens and 4 × 5,000 = 20,000 output tokens. If the orchestrator then reads all four results once, that adds 20,000 input tokens. The subtotal is 60,000 input plus 20,000 output tokens, before the orchestrator’s own brief, answer, reasoning and any further calls. These are total events in the chosen path, not automatically 80,000 tokens of overhead relative to a single agent; a fair baseline must do the same useful work. Briefing and reading the workers are the coordination calls that, as Chapter 8 showed, can erase a cheaper worker’s saving. The workers also start together, so their 40,000 input tokens reach a per-minute rate limit as one burst, and an orchestrator that waits for all four is held up by the slowest.

Cognition, the company behind Devin, argues for a single-threaded agent that compresses its own history, because actions carry implicit decisions and parallel workers that cannot see one another’s choices produce conflicting work.²⁵ The two accounts are closer than Cognition’s title suggests: Anthropic’s report also says most coding tasks have less to parallelize than research.²⁶ I would compare the arrangements on the actual tasks, by cost per accepted task against the same baseline.

Concurrency, retries and held resources

Some costs accrue for as long as a resource is held, used or not. Little’s law, which Chapter 5 applied to load tests, gives the average number held: in a stable system, the average number present equals the arrival rate times the average time each one stays, $L = \lambda W$.²⁷ If a coding agent receives 120 tasks an hour through a sustained busy period and each holds a sandbox for 15 minutes of work, $120 \times 0.25 = 30$ sandboxes are working on average. If one task in five then waits two hours for a reviewer with its sandbox still allocated, those 24 tasks an hour hold another $24 \times 2 = 48$. More sandboxes are waiting than working, and the pool needs headroom above the average because arrivals bunch.

Expiry decides what the waiting costs. Releasing a sandbox and recreating it on resume trades startup time for held capacity, and provisioning one only when a task first needs it, as Anthropic’s managed service does, spares the tasks that never run code.²⁸ The prompt cache lapses during a pause too: resuming the 40-turn task at turn 30 after the five-minute cache has expired costs 68,000 × 1.25 = 85,000 base-rate tokens to rewrite the history, against 6,600 + 2,500 = 9,100 to read 66,000 and write 2,000, and the extra 75,900 is about a quarter of the cached task’s whole input price.

Retries multiply across layers. In the Google SRE book’s example, if the backend, frontend and browser code each make up to four attempts, one user action can become 4 × 4 × 4 = 64 attempts at an overloaded database; the book advises against retrying at several levels and recommends a retry budget with randomized exponential backoff.²⁹ An agent stack adds the model itself, which may decide to try a failed tool call again, and each model-level retry resends the context. If the 40-turn task fails at turn 30 and starts over, it repeats 300,000 + 870,000 = 1,170,000 input tokens, about 60 percent of the task’s input, before doing anything new. A harness that resumes from a durable session log pays at most one cache rewrite. For the agent’s own loop, a step limit does the work of a retry budget.

The operating inventory

Tracing needs its own retention and sampling policy, so consequential actions remain explainable without storing everything forever. OpenTelemetry’s generative-AI conventions, still in development, define spans for agent invocations, tool calls and memory operations, and token attributes that separate cache reads and writes from other input; check which of them a given tracing product captures.³¹

What the review inherits

Failed tasks weigh heavily in that cost because an agent that never converges keeps going until a limit stops it. In the invented loop above, suppose 9,000 of 10,000 tasks succeed in 20 turns (580,000 input tokens each) and 1,000 fail at a 40-turn limit (1,960,000 each). The failures are a tenth of the tasks but 1.96 billion of 7.18 billion input tokens, or 27 percent, and input per accepted task is 7.18 billion ÷ 9,000 ≈ 798,000 tokens rather than 580,000. A 30-turn limit would cut that to 710,000, about 11 percent less, if no task that would have succeeded after turn 30 is lost. Only the distribution of turns to success can show whether that holds.

For economic viability, Anthropic wrote, multi-agent systems need tasks whose value is high enough to pay for the increased performance.³² The workflow owner knows what a resolved ticket or a merged change is worth; the review’s job is to put that value beside the cost and its trend, and to notice when one moves without the other.

When some of this work moves to a model running on hardware the organization rents or owns, the next chapter follows it there.

---

Notes and sources

1. OpenTelemetry, GenAI semantic conventions, checked October 3, 2026; formerly published under opentelemetry.io. Status: Development. Defines agent, tool and memory operations and token attributes for cache reads and writes; support varies by product.↩︎

2. Model Context Protocol, revision 2026-07-28: key changes, transports and tools; and the MCP Registry, in preview; all checked October 3, 2026. Source for the protocol behavior described in this chapter. Implementations adopt revisions at different speeds.↩︎

3. Prateek Chhikara and colleagues, “Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory”, arXiv, April 2025. LOCOMO results (full context: 26,031 tokens, 72.90 percent judged accuracy, 17.117 s at p95; Mem0: 1,764 tokens, 66.88 percent, 1.440 s) and the add, update and delete operations. Vendor-authored; Zep disputes the evaluation in “Is Mem0 Really SOTA in Agent Memory?”, May 6, 2025, updated June 3, 2026.↩︎

4. Zep, Graphiti, checked October 4, 2026. LLM extraction into a bi-temporal graph; superseded facts are invalidated, not deleted; ingestion concurrency defaults to 10 to avoid provider rate-limit errors; Zep’s managed service runs on its own proprietary graph database. Vendor documentation of an open-source framework.↩︎

5. LangChain, “Memory overview”, checked October 3, 2026. The memory taxonomy, following Theodore R. Sumers and colleagues’ “Cognitive Architectures for Language Agents” (2023), and the trade-off between writing memories in the hot path and in the background. Framework documentation, not a measurement.↩︎

6. Prateek Chhikara and colleagues, “Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory”, arXiv, April 2025. LOCOMO results (full context: 26,031 tokens, 72.90 percent judged accuracy, 17.117 s at p95; Mem0: 1,764 tokens, 66.88 percent, 1.440 s) and the add, update and delete operations. Vendor-authored; Zep disputes the evaluation in “Is Mem0 Really SOTA in Agent Memory?”, May 6, 2025, updated June 3, 2026.↩︎

7. Mem0, platform changelog, entry of May 4, 2026, checked October 3, 2026. Describes Memory Decay as a search-time ranking bias that “never filters a candidate out.” Vendor documentation.↩︎

8. Zep, Graphiti, checked October 4, 2026. LLM extraction into a bi-temporal graph; superseded facts are invalidated, not deleted; ingestion concurrency defaults to 10 to avoid provider rate-limit errors; Zep’s managed service runs on its own proprietary graph database. Vendor documentation of an open-source framework.↩︎

9. Charles Packer and colleagues, “MemGPT: Towards LLMs as Operating Systems”, arXiv, 2023. An agent-managed memory hierarchy modeled on virtual memory; the research origin of Letta, whose current design differs.↩︎

10. Letta, “Letta’s Next Phase”, March 16, 2026, and MemFS documentation, checked October 4, 2026. Git-backed memory files; root files load into the system prompt every turn, and every edit is committed. Vendor documentation.↩︎

11. Qdrant, “Capacity Planning”, checked October 4, 2026. Source of the dense-vector and HNSW sizing formulas (m = 16 by default), the default RAM tier, and the advice to quantize “only if the resulting search quality is acceptable.” The guide also adds an ID-tracker term and recommends spare headroom, both left out of the illustration. One engine’s formulas.↩︎

12. Model Context Protocol, revision 2026-07-28: key changes, transports and tools; and the MCP Registry, in preview; all checked October 3, 2026. Source for the protocol behavior described in this chapter. Implementations adopt revisions at different speeds.↩︎

13. Model Context Protocol, revision 2026-07-28: key changes, transports and tools; and the MCP Registry, in preview; all checked October 3, 2026. Source for the protocol behavior described in this chapter. Implementations adopt revisions at different speeds.↩︎

14. Earendil, Pi, checked October 3, 2026, and “Pi Durable”, October 1, 2026. A minimal harness without subagents, described as token-efficient because of its small system prompt (vendor claim); the experimental Pi Durable package checkpoints each step and resumes after a crash.↩︎

15. Anthropic, “Create custom subagents”, Claude Code documentation, checked October 4, 2026. Separate context windows and tool access, shared usage limits, routing to cheaper models, and the 15,000-token description warning. Vendor documentation; limits change between releases.↩︎

16. Anthropic, “Create custom subagents”, Claude Code documentation, checked October 4, 2026. Separate context windows and tool access, shared usage limits, routing to cheaper models, and the 15,000-token description warning. Vendor documentation; limits change between releases.↩︎

17. Nous Research, Hermes Agent, checked October 4, 2026. Open-source harness with local, Docker, SSH and hosted terminal backends; the Daytona and Modal backends hibernate when idle (vendor description). The description is documented capability, not an endorsement of fitness for another organization.↩︎

18. Lance Martin, Gabe Cemaj and Michael Cohen, “Scaling Managed Agents: Decoupling the brain from the hands”, Anthropic, April 8, 2026. The session log sits outside the harness, so a replacement resumes from the last event, and containers are provisioned only when a task needs one. Vendor description of its own service; the Claude Managed Agents overview, checked October 4, 2026, lists the service as in beta.↩︎

19. Earendil, Pi, checked October 3, 2026, and “Pi Durable”, October 1, 2026. A minimal harness without subagents, described as token-efficient because of its small system prompt (vendor claim); the experimental Pi Durable package checkpoints each step and resumes after a crash.↩︎

20. Microsoft, “Set up standard agent resources”, updated July 9, 2026, and “Foundry Agent Service frequently asked questions”, checked October 3, 2026; and “Compare Provisioned Throughput and Serverless”, Azure Cosmos DB documentation, updated April 27, 2026. Basic and standard setups place agent data in different resource arrangements; the standard setup needs at least 3,000 RU/s, multiplied by the number of projects; provisioned throughput is billed hourly regardless of use, serverless by consumption in one region. Classic and new runtimes use different containers; inventory the deployed version rather than copying a fixed resource count.↩︎

21. OpenAI, “Assistants migration guide”, checked October 4, 2026. The Assistants API was sunset on August 26, 2026; earlier history is to be rebuilt from messages the application stored. Vendor documentation.↩︎

22. Anthropic, “Prompt caching” and “Rate limits”, checked October 4, 2026. Five-minute writes at 1.25 times the base input price, one-hour writes at 2 times, reads at 0.1 times (less on some newer models); hits need identical prefixes; on most models, cached reads do not count toward input-tokens-per-minute limits. Provider-specific and subject to change.↩︎

23. Anthropic, “Prompt caching” and “Rate limits”, checked October 4, 2026. Five-minute writes at 1.25 times the base input price, one-hour writes at 2 times, reads at 0.1 times (less on some newer models); hits need identical prefixes; on most models, cached reads do not count toward input-tokens-per-minute limits. Provider-specific and subject to change.↩︎

24. Jeremy Hadfield and colleagues, “How we built our multi-agent research system”, Anthropic, June 13, 2025. Source of the 4× and 15× comparisons with chat in Anthropic’s data, the 50-subagent failure and its scaling rules, the remark on coding tasks, and the value condition. Vendor-published account of one system, not a universal constant.↩︎

25. Walden Yan, “Don’t Build Multi-Agents”, Cognition, June 12, 2025. The case for sharing full context and running a single-threaded agent with history compression; a practitioner’s position, not a controlled result.↩︎

26. Jeremy Hadfield and colleagues, “How we built our multi-agent research system”, Anthropic, June 13, 2025. Source of the 4× and 15× comparisons with chat in Anthropic’s data, the 50-subagent failure and its scaling rules, the remark on coding tasks, and the value condition. Vendor-published account of one system, not a universal constant.↩︎

27. John D. C. Little, “A Proof for the Queuing Formula: $L = \lambda W$,” Operations Research 9, no. 3 (1961): 383–387, doi:10.1287/opre.9.3.383. Holds for long-run averages in a stable system; it says nothing about peaks.↩︎

28. Lance Martin, Gabe Cemaj and Michael Cohen, “Scaling Managed Agents: Decoupling the brain from the hands”, Anthropic, April 8, 2026. The session log sits outside the harness, so a replacement resumes from the last event, and containers are provisioned only when a task needs one. Vendor description of its own service; the Claude Managed Agents overview, checked October 4, 2026, lists the service as in beta.↩︎

29. Mike Ulrich, in Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy, eds., Site Reliability Engineering (O’Reilly, 2016), Chapter 22, “Addressing Cascading Failures”, checked October 3, 2026. Source of the 4³ = 64 example, the advice against retrying at multiple levels, and the retry budget. Written about serving systems; applying it to agent loops is the author’s reasoning.↩︎

30. Anthropic, “Prompt caching” and “Rate limits”, checked October 4, 2026. Five-minute writes at 1.25 times the base input price, one-hour writes at 2 times, reads at 0.1 times (less on some newer models); hits need identical prefixes; on most models, cached reads do not count toward input-tokens-per-minute limits. Provider-specific and subject to change.↩︎

31. OpenTelemetry, GenAI semantic conventions, checked October 3, 2026; formerly published under opentelemetry.io. Status: Development. Defines agent, tool and memory operations and token attributes for cache reads and writes; support varies by product.↩︎

32. Jeremy Hadfield and colleagues, “How we built our multi-agent research system”, Anthropic, June 13, 2025. Source of the 4× and 15× comparisons with chat in Anthropic’s data, the 50-subagent failure and its scaling rules, the remark on coding tasks, and the value condition. Vendor-published account of one system, not a universal constant.↩︎