The Invisible Trillion · The Energy Question, Honestly

Architecture, Not Amount

Connected to the above point but worth its own treatment: the energy cost of AI depends much more on where the inference runs than on how much of it there is.

Consider a specific task — say, summarizing an email — and trace the energy cost under different architectures.

If the summary runs through a frontier-model API: your phone sends the email text to a hyperscale data center, which loads the email into a model running on $40,000 GPUs cooled by industrial-scale chillers, in a building drawing 100 megawatts of power, on a grid that has been expanded to serve that data center at residential ratepayer expense. The inference itself takes perhaps half a second of GPU time, consuming a few watt-hours. The total energy cost of the inference, properly accounted, includes the share of the data center’s idle-but-running infrastructure, the share of the cooling load, the share of the grid expansion. The end-to-end energy might be ten to twenty watt-hours, depending on how you count.

If the summary runs on a 4-billion-parameter model on your phone: your phone never sends the email anywhere. The model, already loaded into the phone’s neural processing unit, runs the inference in a few hundred milliseconds, drawing perhaps 0.1 watt-hours from the phone’s battery — energy that came from the wall outlet you plugged the phone into last night, at residential rates, with no marginal grid expansion required. The end-to-end energy cost is one one-hundredth of the cloud version, and you didn’t ask any utility or any data center to add capacity to serve you.

Same task. Two architectures. Two orders of magnitude difference in energy cost. The difference is not in how good AI is or how much people use it. It is in where the inference runs.

The chapter on self-driving — which comes a few chapters later in the book — describes the same architecture argument in more detail. Self-driving was forced onto an edge-inference architecture by the physical constraints of real-time driving. The result is that the world’s fleet of partially and fully autonomous vehicles, running billions of inference operations per day, has an energy footprint that mostly does not show up in hyperscale data center demand. Each car’s compute runs on the car’s battery, paid for at the rates the car owner pays for charging, with no centralized grid expansion required.

Route general inference the way self-driving routes it, and the demand-growth projections shrink dramatically. The “rising fast” trajectory is partly real growth and partly an artifact of sending inference to hyperscale data centers that did not need to handle it — a business-model preference on the providers’ part, not a technical necessity, whose energy cost is paid by ratepayers and host communities.

All 19 chapters of The Invisible Trillion · Library · All courses