Quick summary · Chapter 7
AI Services and Token Capacity
First editionThe essential ideas, decisions, and diagrams from this chapter. Follow the section links for the examples, calculations, and fuller argument.
First-edition visual companion · Selected from the current chapter · 8x10-inch PDF
Before any theory: the four ways to buy tokens
There is more than one way to buy the same token, and the differences matter to a capacity plan.
The most direct path is the provider’s own API, usually priced by consumption: so much per million input tokens, so much per million output tokens, with separate treatment for caching and other features. Payment may be prepaid, invoiced, or governed by a contract; a usage-based price does not establish when the money changes hands.12
The second path is through a platform. Microsoft Foundry, Google’s Gemini Enterprise Agent Platform (formerly Vertex AI) and OpenRouter provide model access under their own deployment, routing and billing arrangements.345 Before comparing rates for a named model, establish which version and features each offering supports, who supplies the inference, where processing may occur and which agreement pays for it. An existing cloud agreement can make this path operationally convenient; the workload still needs its own comparison.
The third path is a negotiated agreement. Large consumers may negotiate discounts, usage minimums, and model-specific terms. Negotiation does not guarantee a discount or held supply, and the price of the same work can depend on which agreement it flows through.
The fourth path is provisioned capacity, because a workload can hit a throughput limit while its monthly spending is modest. Providers sell arrangements that reserve a defined level of processing capability, subject to the model, region, workload and service terms, and the charge continues while that capacity is idle. A spending commitment alone reserves no capacity.67
Token capacity is a different kind of capacity
What limits an AI service is a combination of three things that no single graph shows:
- Scarcity. Rate limits cap requests, tokens or concurrency within a window, and their scope varies by model and account. On shared service a limit is permission to send rather than capacity held for the sender, so a request within it can still be refused when the provider is busy. Queuing and retries are application decisions with latency and cost consequences.8
- Pricing. Per-token charges, subscriptions and provisioned capacity respond differently to demand. Compare the full cost of the service, including unused commitments, before drawing a conclusion from its token rate.
- Budgets. A prepaid balance can run out; a commitment can charge after demand disappears. Alerts, hard caps and advisory limits behave differently, so the plan needs to say which one is in force and what happens when it is reached.
The model product list and how to read it
Sizing supply for the busy interval
Measure the efficiency of the models, not just the spend
Consumption efficiency counts accepted tasks per million tokens:
Record it per model and per task class, measured monthly.
Accepted ties the count to the business’s outcome standard; preserve whether the result needed correction and what that correction cost. Per task class keeps unlike work from being averaged together. Model-level comparisons are useful when a model performs the complete task. For a workflow spanning several models, report the end-to-end outcome once and use step-level costs and acceptance checks to explain each model’s contribution. Crediting every model with the same outcome and then summing would count the work repeatedly.
Recommended practices
Size supply for the work that must finish on time. Use measured demand over the relevant interval, the permitted delay and an explicit uncertainty allowance. Decide which work may queue, which may use overflow and which needs reserved throughput, and test any reservation on price as well as supply.
Run the efficiency measurement every month, with an explanation. Report accepted tasks and token consumption by model and task class, alongside total cost and retries. Investigate changes rather than assuming they represent improvement or waste.
Keep the budget policy usable across models and providers. State who may spend, what is counted and what happens at the limit separately from the feature implementing it. A provider change may require different controls: a balance, gateway quota, spend cap and commitment make different promises. Re-test their behavior rather than transferring the old configuration by name.
OpenRouter, FAQ, https://openrouter.ai/docs/faq, and “API Credit & Rate Limits,” https://openrouter.ai/docs/api_reference/limits, checked October 4, 2026; Stripe, “Stripe agrees to acquire OpenRouter to help businesses optimize token routing and usage,” August 19, 2026, https://stripe.com/newsroom/news/stripe-agrees-to-acquire-openrouter. Product terms and transaction announcement; no transaction price is inferred.↩︎
OpenAI, “GPT-6 Astra: A new generation of intelligence,” September 3, 2026, https://openai.com/index/gpt-6-astra/. Launch Standard rates $10/$50 per million input/output tokens; Fast mode twice the Standard price and advertised at up to twice the speed. On OpenRouter, which reports weekly route traffic, Astra was among the most-used routes through early October 2026 and processed about 98.6 billion tokens on October 9, 2026, moving down the popularity list.↩︎
Microsoft, “Understanding deployment types in Microsoft Foundry Models,” https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/deployment-types, checked October 3, 2026. Deployment categories, supported processing boundaries and model-specific availability; managed-compute deployments are a separate option.↩︎
Google Cloud, “Use Provisioned Throughput,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/provisioned-throughput/use-provisioned-throughput, checked October 4, 2026. Default spillover, the dedicated-only 429, enforcement windows and spillover hidden by one-minute averages. Renamed from Vertex AI on April 22, 2026, per the release notes: https://docs.cloud.google.com/gemini-enterprise-agent-platform/release-notes.↩︎
OpenRouter, FAQ, https://openrouter.ai/docs/faq, and “API Credit & Rate Limits,” https://openrouter.ai/docs/api_reference/limits, checked October 4, 2026; Stripe, “Stripe agrees to acquire OpenRouter to help businesses optimize token routing and usage,” August 19, 2026, https://stripe.com/newsroom/news/stripe-agrees-to-acquire-openrouter. Product terms and transaction announcement; no transaction price is inferred.↩︎
Microsoft, “Provisioned throughput billing and cost management,” https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/provisioned-throughput-billing, checked October 4, 2026. Deployed throughput incurs hourly charges whether or not it is used; a reservation is a discount on those charges and does not guarantee capacity.↩︎
Google Cloud, “Use Provisioned Throughput,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/provisioned-throughput/use-provisioned-throughput, checked October 4, 2026. Default spillover, the dedicated-only 429, enforcement windows and spillover hidden by one-minute averages. Renamed from Vertex AI on April 22, 2026, per the release notes: https://docs.cloud.google.com/gemini-enterprise-agent-platform/release-notes.↩︎
OpenAI, “Rate limits,” https://developers.openai.com/api/docs/guides/rate-limits, and “Error codes,” https://developers.openai.com/api/docs/guides/error-codes; Google Cloud, “Standard PayGo,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/deploy/standard-paygo, and “Priority PayGo,” https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/deploy/priority-paygo; Microsoft, “Enable priority processing for Microsoft Foundry Models,” https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/priority-processing; AWS, “Service tiers for optimizing performance and cost,” https://docs.aws.amazon.com/bedrock/latest/userguide/service-tiers-inference.html; checked October 3, 2026. Documented ramp, retry, spend-limit and downgrade behavior; shared pool sizes are not published.↩︎