The Invisible Trillion · How Much Do We Actually Need?

Where The Inference Should Run

Now to the part of the analysis that most directly addresses the inflation hypothesis. Inference — 80 to 90% of all AI compute — can run in many places, and the choice of where has dramatic implications for how much infrastructure is actually required.

There are roughly three places inference can run:

On the edge — meaning on the device the user is already holding. A phone, a laptop, a tablet, a smart speaker, a car, a robot. The device runs the model locally. No data leaves the device. No cloud round-trip happens. The latency is near zero. The cost to the user is whatever battery and compute the device uses to run the model. The cost to the provider is zero, because the provider is not providing any compute.

On nearby compute — meaning a local server, an edge data center near the user, or a small regional cloud presence. Latency is low. Data may or may not leave the user’s immediate environment. The compute is paid for by whoever runs the local server, which may be the user, the user’s employer, or a regional cloud provider.

On hyperscale cloud — meaning a frontier-model API call to OpenAI, Anthropic, Google, or one of their hyperscale competitors. The data leaves the device, is sent to a hyperscale data center somewhere (possibly thousands of miles away), processed by a very large model running on very expensive infrastructure, and the result is sent back. Latency is hundreds of milliseconds to seconds. The cost per query is meaningful — modern frontier APIs charge on the order of $1–15 per million tokens, depending on the model and the task.

The choice among these three depends on the task. Some tasks genuinely require hyperscale cloud — frontier-level reasoning, very large context windows, multimodal generation, real-time access to up-to-date information. Some tasks can be perfectly well served on the edge — autocomplete, simple classification, transcription, basic chat, photo organization. Some tasks fall in between and benefit from hybrid architectures where simple parts run locally and complex parts route to the cloud.

The technical question is which tasks belong where. The economic question is which routing decisions get made, and why.

What Edge Compute Can Actually Do In 2026

The capabilities of edge devices have improved dramatically in the past three years, and most people — including most people in the AI industry — have not internalized how much. Some specific 2026 numbers.

Phones. A flagship 2026 phone — iPhone 17 Pro, Pixel 10, Samsung Galaxy S26 — has a neural processing unit (NPU) capable of 40–80 trillion operations per second (TOPS). This is enough to run language models with 1–3 billion parameters at reading speed, comfortably. A model like Microsoft’s Phi-4-mini (3.8 billion parameters) or Meta’s Llama 3.2 (3 billion parameters) or Google’s Gemini Nano runs locally on these phones with no cloud connection required.

What can a 3-billion-parameter model do? In 2023, the answer was “not much.” In 2026, it can do most of what 95% of users actually use AI for: rephrasing emails, summarizing documents, answering basic factual questions, performing simple translations, writing short text, doing basic reasoning, generating code snippets, providing tutoring on standard educational topics. The 2026 small models, trained on large amounts of high-quality data, are dramatically more capable than the 2023 small models of the same size. They are not as good as frontier models for the hardest tasks. They are entirely adequate for the median task.

Laptops. A 2026 laptop with modern silicon — Apple M4 or M5, Qualcomm Snapdragon X2, Intel Lunar Lake, AMD Ryzen AI 300 — can run models of 7–14 billion parameters comfortably, and models up to 70 billion parameters with substantial system memory. An Apple M4 Max with 128 gigabytes of unified memory can run a 70-billion-parameter model at INT4 quantization at usable speeds, locally, with no cloud connection. The cost of inference on this hardware, amortized over the laptop’s lifetime, is approximately $0.014 per million tokens. For comparison, a frontier API call to Claude or GPT-4 class models in 2026 costs $3–15 per million tokens depending on the model and the input/output ratio.

The cost differential is three orders of magnitude. Running inference locally on a laptop the user already owns costs approximately one one-thousandth of what running the same inference through a frontier API costs.

Mid-range hardware. A modest desktop with a consumer GPU (RTX 4060, 4070, 4080, equivalent AMD) can run 7–30 billion parameter models comfortably. A workstation with multiple GPUs or a Mac Studio with substantial memory can run frontier-adjacent models locally. None of this is exotic hardware. It is hardware that millions of small businesses, professional users, and serious hobbyists already own for other reasons.

Quality of small models in 2026. Here is the part most people miss. The small models in 2026 are not the small models of 2023. They have been trained on much larger amounts of data, with much better training recipes, with much better fine-tuning, and with techniques like distillation that compress the capabilities of larger models into smaller ones. A 4-billion-parameter model in 2026 substantially outperforms a 70-billion-parameter model from 2023 on most benchmarks. The state of the art for small models has moved much faster than most people realize.

The practical implication is this: for the substantial majority of AI use cases an ordinary person or ordinary business has, edge compute in 2026 is already adequate. Not someday adequate. Not adequate-with-some-caveats. Adequate. Right now. The model running locally on your phone is good enough to do what you actually want done, in most cases.

What Hyperscale Cloud Is Genuinely Required For

This does not mean hyperscale cloud is unnecessary. There are categories of work where it is genuinely required, and being honest about which categories those are is part of evaluating whether the buildout is justified.

Frontier model training. Discussed above. Real, large, but rare.

Very long context reasoning. Some tasks require processing hundreds of thousands or millions of tokens of context — analyzing an entire codebase, summarizing a year of email, reasoning over a large legal document corpus. Models that can handle these context lengths efficiently currently require hyperscale-class hardware. Edge devices, in 2026, can handle contexts up to perhaps tens of thousands of tokens efficiently. Beyond that, hyperscale is currently required.

State-of-the-art reasoning on hard problems. For the hardest problems — competition mathematics, novel scientific reasoning, complex multi-step planning — frontier models that are substantially larger than what fits on edge devices remain measurably better. The gap is closing, but it is real, and for users whose work depends on the absolute frontier (researchers, advanced developers, certain professional applications), the gap matters.

Multimodal generation at high quality. Generating images, video, or audio at production quality currently requires substantial compute. Edge devices can do real-time image generation at modest quality and small resolutions; producing the kind of output that matches dedicated graphics pipelines still requires cloud infrastructure.

Real-time access to up-to-date information. Edge models have a knowledge cutoff and cannot search the web. Tasks that require current information — news, prices, weather, recent events — require some form of connectivity, though the connectivity could in principle route to a small retrieval system rather than a frontier model.

Shared services with low individual usage. Some tasks are used rarely by any individual but used often across the population. Translation between obscure language pairs. Specialized medical diagnostic tools. Niche professional applications. Hosting these centrally, rather than installing them on every device, is more efficient.

Genuine multi-user coordination. Real-time multiplayer game AI, collaborative document editing, large-group conversation summarization — anything where the compute genuinely needs to coordinate across multiple users.

Privacy-preserving but compute-intensive workflows. Some workflows benefit from centralization for non-obvious reasons — federated learning, where models are improved using data that never leaves user devices, still requires central coordination of the aggregation step.

This is the list of things hyperscale cloud is genuinely required for. It is not nothing. It is also, in aggregate, much smaller than the current buildout assumes. The list above does not justify three hundred new hyperscale data centers per year. It justifies maybe a third of one year’s worth of current buildout, sustained at current capacity.

What Hyperscale Cloud Is Being Used For That Doesn’t Belong There

Now consider the inverse. Look at the actual inference traffic currently routed to hyperscale cloud APIs, and ask which fraction of it genuinely requires hyperscale-class infrastructure.

The honest answer is: a substantial fraction does not. A substantial fraction of the queries currently routed to frontier models are queries that a 4-billion-parameter local model would handle adequately. Email rephrasing. Document summarization. Basic question-answering. Most coding assistance. Most customer service. Most translation. Most autocomplete. Most photo and image processing. Most personal assistant tasks. None of these benefit meaningfully from frontier-model capability over what a competent small model can do, and most of them would have lower latency, better privacy, and dramatically lower cost on the edge.

The reason they currently route to frontier models is partly inertia (the integrations were built that way), partly user-experience uniformity (the frontier model handles every case adequately even when most cases don’t require it), and partly economic incentive on the part of the providers (each frontier-model call generates revenue; each local-model call does not). The economic incentive is the largest single factor. If frontier-model API providers had to compete head-to-head with local-model inference on cost and latency for the median task, they would lose, decisively, and the cloud-side infrastructure required would collapse.

This is the structural feature that makes the inflation hypothesis plausible. The current architecture, where most AI inference routes through frontier-model APIs running on hyperscale infrastructure, is not a technical optimum. It is a commercial optimum for the providers. The shift toward edge inference — which would dramatically reduce hyperscale infrastructure requirements — is being slowed not by technical limits but by business model considerations. The providers do not want it to happen, and they have the resources to slow it down.

The shift is happening anyway, slowly, despite the providers. 42% of developers were running LLMs locally as of early 2026, up from a small minority two years earlier. The tooling is improving. The models are getting better. The hardware is getting more capable. The economic pressure from frontier API pricing is pushing users to find alternatives. The trajectory is clearly toward more inference on the edge and less in the cloud. The trajectory implies that the current hyperscale infrastructure is being built for a demand pattern that is going to decay before the infrastructure is paid off.

This is not a fringe view. The VentureBeat survey of IT decision-makers found that “cost per inference / total cost of ownership” jumped from 34% to 41% as a top priority in a single quarter in early 2026. Enterprises are starting to migrate workloads off frontier APIs and toward smaller, cheaper, often local alternatives, because the unit economics of running production systems on frontier APIs do not work. The infrastructure being built today is not being sized for the demand pattern of 2027 and 2028, when the migration will be further along. It is being sized for the demand pattern of 2024 and 2025, projected forward as if it would continue indefinitely.

There is a deeper version of this point, and it is the one that most directly inflates the numbers. The argument so far has been about model size — that a small model would handle the median task as well as a frontier one. But a meaningful slice of what gets routed to a frontier API is not a language task at all. It is deterministic work with a single correct answer: looking a value up in a database, converting a file from one format to another, classifying a record, routing a request, pulling a field out of a form.

A frontier model is a probabilistic machine — it returns the likely continuation of its input, not a guaranteed-correct one — and sending deterministic work to it is like convening a jury to decide whether two plus two is four. Enormous overhead, real per-call cost, and a small but nonzero chance the jury returns the wrong number. A plain database query does it faster, cheaper, and correctly every single time.

This work migrates downward fastest of all, because it never needed a model to begin with — only the early-adoption reflex of reaching for the most capable tool by default, before the bill arrived. As it moves to small models, to distilled models, and to ordinary deterministic code, the demand it represented does not so much shrink as disappear: it was never durable demand for frontier inference in the first place.

This is what speculative buildouts look like in their late stages: capacity additions made on the assumption that current usage patterns will persist, while the underlying conditions that created those usage patterns are shifting.

All 19 chapters of The Invisible Trillion · Library · All courses