Energy & compute · Primer
Every economic question in AI — capex sustainability, token pricing, GPU-backed credit, compute derivatives — reduces to a physical one: what silicon is doing the work, how efficiently it is being used, and who bears the price risk on it. This is the chain, written for a markets and risk audience.
LLM compute divides into three workloads with very different hardware demands.
Training is a massive parallel matrix-multiplication job. It is compute-bound, runs for weeks to months across thousands to hundreds of thousands of interconnected accelerators, and is intolerant of failure. It rewards raw arithmetic, high-bandwidth memory, and — critically — the interconnect fabric that lets tens of thousands of chips behave like one machine.
Inference — prefill processes the input prompt. It is also compute-bound: the prompt’s tokens can be processed in parallel through large matrix multiplications, at intensities of roughly 100–400 arithmetic operations per byte moved. Prefill sets time-to-first-token, the latency a user feels before anything appears.
Inference — decode generates the answer one token at a time, and two pieces of stored data explain its entire economics. The weights are the model itself — the tens or hundreds of billions of numbers learned during training, typically hundreds of gigabytes sitting in the chip’s memory. Think of them as the model’s rulebook. Producing a token means multiplying the current state of the conversation through that whole rulebook, so all of the weights must be read out of memory for every single token. The KV cache is the model’s working memory of the conversation so far: to stay coherent, each new token must compare itself against every token before it, and rather than redo that arithmetic each step the chip saves the intermediate results. That saved pile accumulates as the conversation lengthens, which is why long contexts are disproportionately expensive to serve.
So each output token requires hauling the whole rulebook plus the whole conversation file out of memory, doing a comparatively trivial amount of arithmetic on them — on the order of one to two operations per byte moved — and repeating. Decode is bounded by how fast data can move, not how fast the chip can calculate.
Reading the prompt is a math problem. Writing the answer is a data-moving problem. A GPU is built with both a very large calculator and a very fast memory pipe, and each job stresses only one of them. While the chip writes the answer the calculator sits nearly idle waiting for data; while it reads the prompt the memory pipe is underworked.
An H100 SXM peaks around 989 TFLOPS of dense FP16/BF16 tensor throughput. Roofline arithmetic on its 3.35 TB/s of memory bandwidth suggests a decode workload may realise only single-digit to low-tens of TFLOPS at small batch sizes — an estimate rather than a published measurement, but the direction is the point. The owner paid for both capabilities on every chip and each phase uses only one at a time. That wasted capacity is the economic opening every specialized-chip strategy is aimed at.
Output tokens price at roughly 4× input tokens at most API providers, because decode consumes far more scarce memory bandwidth per token. And reasoning models multiplied the decode share: chain-of-thought and agentic workloads generate an order of magnitude or more output tokens per query than 2023-era chat. Inference is now commonly estimated at around two-thirds of all AI compute — Bloomberg Intelligence and Deloitte both put it near that mark — and the mix within inference has shifted hard toward the memory-bound phase.
Put those together. Revenue is per-token; output tokens carry the premium price and dominate volume, so revenue concentrates in decode. Cost concentrates there too, because a GPU generating output tokens produces far fewer billable tokens per hour than one crunching prompts. Gross margin collapses to a spread — the market price of an output token minus your cost to run decode — and the operator with the cheapest decode wins. Prefill quality shows up in user experience; decode efficiency shows up in the P&L.
A note on names: Nvidia names each generation after a scientist — Hopper, Blackwell, and next Rubin, after the astronomer Vera Rubin. Hopper (H100/H200) is the deployed installed base; Blackwell (B200/GB200/GB300) is the current volume generation. Rubin was launched at CES in January 2026 for second-half 2026, with Nvidia claiming up to 5× inference performance and 10× lower cost per generated token versus Blackwell at rack level. Those are vendor claims for favourable workloads; independent analysts put dense-transformer gains closer to 2–3×, and the distinction matters enormously for the forward curves in the appendix.
In December 2025 CNBC reported that Nvidia was buying Groq’s assets for about $20 billion, which would have been its largest deal ever. Nvidia publicly denied acquiring Groq, describing instead a non-exclusive licence to certain Groq intellectual property together with the hiring of engineering talent, including founder Jonathan Ross. Groq’s own statement said it continues as an independent company under a new chief executive and that its cloud business was not part of the transaction. Reporting remains contested and the CNBC figure traced to a third-party investor rather than either company.
The strategic logic is worth understanding regardless of the corporate form, because it follows directly from the decode mechanics above. Groq’s chip holds model data in SRAM — memory etched onto the chip itself, tiny in capacity but roughly an order of magnitude faster per byte than the stacked HBM bolted alongside a GPU. Since decode is precisely a memory-speed problem, an SRAM-based design generates output tokens fast and cheap while consuming none of the scarce HBM supply. A division of labour follows: prompt-reading on the best calculators, answer-writing on purpose-built data movers, with the second-order benefit that every decode workload moved off HBM frees that constrained input for training, where SRAM’s small capacity rules it out and HBM has no substitute.
AMD competes on paper specs and price with the MI350/MI355X, with the MI400 series and the ROCm software stack as swing factors. It remains the credible second source that keeps Nvidia’s pricing honest.
Google’s TPU is the largest non-Nvidia fleet. TPU v7 “Ironwood” delivers 4,614 TFLOPS FP8 with 192 GiB of HBM at 7.37 TB/s, deployed in pods of 9,216 chips over optical interconnect. The strategically important shift is that TPUs are no longer internal-only: Google announced in October 2025 that Anthropic would have access to up to one million TPU chips and well over a gigawatt of capacity coming online in 2026 — a contracted ceiling rather than an operating fleet, and worth stating precisely, since the difference between contracted access and deployed silicon is exactly the sort of distinction that matters for supply modelling.
Amazon’s Trainium 3 offers 2.52 PFLOPS FP8 and 144GB of HBM3e at 4.9 TB/s, in UltraClusters scaling to a million chips, with AWS claiming roughly 50% lower training and inference cost versus GPU alternatives. Microsoft’s Maia 200, announced January 2026, delivers over 10 PFLOPS FP4 and 216GB of HBM3e at 7 TB/s on a 3nm process. Meta’s MTIA line serves ranking and recommendation plus growing internal LLM inference. And OpenAI’s collaboration with Broadcom, announced October 2025, targets 10GW of deployed custom accelerator capacity with deployment beginning in the second half of 2026 and completing by the end of 2029 — the clearest signal that every frontier lab intends to own part of its silicon destiny.
Model FLOPs Utilization — the share of a chip’s theoretical arithmetic capacity actually converted into useful model computation — is the single most important operational metric in compute economics. Well-run training jobs achieve 35–50%; naively served inference can sit in single digits, meaning the operator gets less than a tenth of the horsepower they paid for. The gap between purchased capacity and delivered tokens is where the money leaks, and an entire software industry exists to close it.
Decode’s cost is hauling the weights out of memory for every token, and that haul costs the same whether the chip is answering one conversation or forty. So serve forty at once and one trip through the weights produces forty billable tokens instead of one: a bus instead of forty taxis. “Continuous” is the scheduling refinement — instead of making the bus wait until every passenger’s trip ends, finished requests hop off and new ones hop on at every stop.
Each conversation’s working-memory file lives in scarce chip memory, and early serving systems reserved one contiguous block per conversation sized for the longest possible answer, most of which sat empty. Paged attention borrows a fifty-year-old operating-system trick: chop the cache into small pages and allocate them only as the conversation grows. Less waste per conversation means more conversations resident per chip, which feeds directly into batch sizes.
Generating tokens one at a time is expensive, but checking a proposed sequence is cheap, because verification parallelises like prefill. Pair the big model with a small draft model that guesses the next several tokens; the big model validates the whole guess in one pass and keeps whatever was right. Since most next words are easy to predict, the guesses usually survive, and the expensive model makes one memory haul per several tokens instead of one per token.
Weights do not need full precision to work. Rounding them to a coarser format halves the bytes that must move. Decode is billed by the byte moved, so halving the bytes roughly doubles speed and halves the memory footprint, and quality holds up better than intuition suggests. This is the closest thing the industry has to a free lunch, which is why each hardware generation adds support for coarser formats.
The prefill/decode split made operational: route the math-shaped work and the memory-shaped work to different chips, so neither wastes the other’s capacity. Even within a single GPU the scheduling insight pays. Chunked prefill slices one user’s enormous document into pieces and interleaves them between other users’ token-generation steps, so nobody’s streaming answer freezes because someone else uploaded a three-hundred-page PDF. Published gains vary by workload mix; the Sarathi-Serve work reports up to 2.6× serving capacity within a latency target, while measured throughput improvements on other mixes are more modest.
None of these techniques requires buying a single additional chip. The same fleet delivers wildly different token output depending on the software driving it, which makes serving-stack quality — not hardware ownership — the real dispersion factor in operator economics.
For a markets reader the analogy is capacity factor in power generation: the same nameplate asset is a great or terrible investment depending entirely on realized utilization, and utilization here is a software variable. This is why identical GPUs command wildly different effective economics across operators — and why defining “a unit of compute” for a settlement benchmark is a genuinely hard indexing problem.
Capex. Microsoft, Alphabet, Amazon, Meta and Oracle have collectively guided to $660–690 billion of 2026 capital expenditure. The share directed specifically at AI is widely quoted at around three-quarters, though that split traces to analyst estimates rather than company disclosure.
Fab capacity. Every major 2026 accelerator fabricates on TSMC 3nm-class processes, and 3nm utilization has run at capacity through 2026 with output rising toward 180,000 wafer starts a month by early in the fourth quarter and still not meeting demand. Advanced packaging and HBM supply are the recurring pinch points beneath the headline.
Power. The binding constraint has migrated from chips to electrons. Rack-scale systems now run above 100kW per rack, forcing liquid cooling and campus-scale power engineering, against multi-year grid interconnection queues. Estimates of how much announced US capacity will actually energize vary sharply and are contested — one widely cited analysis suggests a large fraction of 2026’s planned gigawatts will slip or cancel, and at least one specialist shop has published a direct rebuttal of that reading. What is not contested is the utility response: investor-owned utilities have capital plans above $1.1 trillion for 2025–2029. For allocation this means compute increasingly gets deployed where power is, not where demand is.
Allocation between training and inference. Fleets are actively re-partitioned: newest, best-interconnected silicon goes to training, while prior-generation and specialized silicon cascades to inference. That cascade gives older GPUs a second economic life — and it is precisely the assumption embedded in every depreciation schedule and every GPU-collateralized loan.
A neocloud installs an H100 in early 2024 at roughly $40,000 all-in — about $28,000 for the chip plus its share of networking, cabling and facility fit-out — financed largely with debt and depreciated straight-line over six years, about $6,700 a year of book depreciation.
Training tenancy, years 1–2. A frontier lab rents the cluster on a two-year contract at $4.00 per GPU-hour. At 85% utilization that is roughly $30,000 a year of revenue against cash operating cost near $4,000. Two years recovers $52,000 against a $40,000 box. This is why 2023–24 vintage deals looked extraordinary, and why lenders lent.
The cascade, year 3. Blackwell arrives and the training tenant migrates, because frontier training wants the new generation’s interconnect and will pay for it. The H100 redeploys to inference: shorter jobs, lumpier demand, utilization down to about 60%, and a market rate repriced toward $2.00. Revenue falls to roughly $10,500. The asset is still solidly cash-positive — the second economic life is real.
Decay, years 4–6. Each subsequent generation pushes the chip further down-market into fine-tuning, small-model serving and batch inference where latency does not matter. An illustrative rate path of $1.60, $1.30 and $1.00 gives steadily thinning cash margins.
Shutdown. The GPU keeps running as long as spot rental exceeds its variable cash cost. Below that, running it burns money and capacity withdraws — the identical logic to a power plant retiring when its spark spread goes negative. This is the real supply discipline in the rental market.
GPT-4 launched in March 2023 at $30 per million input tokens and $60 per million output. GPT-4-level capability is now available well under a dollar per million. The rate of decline most often quoted is roughly 10× a year for constant capability — a16z’s estimate, and the conservative end of the published range; Epoch AI measures between 9× and 900× a year depending on the benchmark, with GPT-4-level science questions around 40×.
Token prices falling that fast while aggregate compute spend rises is not a contradiction. It is the defining feature of the market: cheaper tokens unlock demand — agents, reasoning, video, code — faster than prices fall, so far. The open risk question is whether that elasticity holds at every point in the cycle, and the levered parts of the capital structure are implicitly short the scenario where it doesn’t.
As prefill and decode split onto different silicon, three structural consequences follow. The HBM oligopoly loses some pricing power as decode migrates toward SRAM-based architectures. The orchestration layer — deciding which phase of which request runs on which chip — becomes a margin control point in its own right. And capital intensity per delivered token falls, opening defensible niches for non-Nvidia silicon precisely where the market is growing fastest.
The industry finances long-lived infrastructure — five-to-six-year depreciation, fifteen-to-thirty-year power assets, twenty-year lease guarantees — against a revenue stream whose unit price falls roughly an order of magnitude a year and whose hardware basis turns over on what is now an annual product cadence. Bridging that mismatch is the central financial-engineering problem of the era.
Michael Burry’s November 2025 critique crystallized the issue: he estimated that five hyperscalers would understate depreciation by roughly $176 billion across 2026–2028 by carrying useful lives longer than the product cycle, with implied earnings overstatements he put in percentage terms by company. The deeper point is not the accounting but the rental curve. If rental rates keep collapsing on each generation, residual values fail regardless of utilization.
The majors themselves disagree. Amazon shortened a subset of servers to five years effective January 2025, expecting a roughly $700 million operating-income hit. Meta extended to five and a half years, a $2.9 billion depreciation reduction. Microsoft, Alphabet and CoreWeave sit at six. Every one of those disputes is an argument about the slope of Phases 2 and 3 in the figure above, and every GPU-collateralized loan’s loan-to-value is marked against exactly that curve.
H100 spot rentals fell from above $7 an hour in early 2024 into a $2–4 band through 2025 — and then partially reversed. Nvidia raised H100 rental pricing in 2026, the indices recorded a spike, and August 2026 quotes span roughly $2.19 to $11.06 depending on tier. Legacy supply stopped growing as wafers moved to the new generation, just as inference demand accelerated. Anyone modelling residual value off a smooth decline is modelling the wrong shape.
An often-cited estimate puts GPU-collateralized debt above $20 billion, though it is a press figure without a primary tally and is probably stale on the low side. What is precisely documented is the price. CoreWeave’s $2.6 billion first-lien delayed-draw term loan — priced in late July 2026 after flexing wider from initial talk, closed August 10 — cleared at SOFR+550 at an issue price of 97, a 10.44% yield to maturity, rated Ba2/BB+, arranged by JPMorgan and MUFG, and carrying maintenance covenants including a 1.35× debt-service coverage ratio at a financing subsidiary. Contracted offtake plus GPU collateral was not enough to avoid them.
Into that gap stepped the vendor. Nvidia signed non-binding memoranda in August 2026 with six large asset managers to establish financing platforms intended to mobilize over $500 billion of third-party capital over time. It separately agreed in September 2025 to purchase up to $6.3 billion of CoreWeave’s unsold capacity through April 2032, and in August 2026 guaranteed up to $105 billion of conditional lease and power obligations to SB Energy supporting OpenAI’s twenty-year lease at a campus in Pike County, Ohio.
The manufacturer of the asset is now also, in substance, the residual-value insurer of the asset. Economically that is elegant: when old GPUs lose value faster, new-generation sales accelerate, so the chipmaker is structurally short obsolescence risk in its core business and can warehouse the long side more cheaply than anyone.
It is also exactly the reflexive pattern — vendor financing supporting demand for the vendor’s own product — that risk committees flagged in telecom in 1999 and in merchant energy in 2001. And protection written by the entity whose own product launch is the trigger event has obvious wrong-way properties. The companion piece works through who else could write it.
In the current expansion, fabs and HBM makers enjoy allocation power. Chip designers capture scarcity rents and rotate margin into ecosystem defence. Hyperscalers convert balance-sheet strength into fleet scale and custom-silicon cost curves, and their ASIC programs are the structural threat to merchant pricing. Neoclouds lever contracted offtake into rapid fleet growth, and their spread — long depreciating hardware, short falling rental rates — is the cycle’s most exposed book. Power and land owners capture an increasing share of rents as the constraint migrates to electrons.
In a repricing phase the transmission channel runs rental rates to collateral values to neocloud refinancing to vendor backstops to a chip-order air pocket. Lenders discover recovery values on three-year-old accelerators; insurers of residual value discover correlation, because every insured GPU reprices on the same launch day; utilities discover which interconnection commitments were real. Downstream, application companies are the clear countercyclical winners — compute deflation is their margin tailwind in every scenario. The players who fare best across the whole cycle are those holding priced, hedged or contractually transferred exposure rather than implicit spread risk.
Export controls. The earlier diffusion rule was rescinded in 2025 and replaced by a transactional, case-by-case regime. A BIS final rule effective January 15, 2026 moved H200 and MI325X-class exports to China from presumption of denial to case-by-case review, subject to performance ceilings — total processing performance under 21,000 and total DRAM bandwidth under 6,500 GB/s — plus a cap requiring aggregate China shipments not exceed 50% of units shipped to US customers, supply-preservation and no-diversion certifications, consignee know-your-customer covering remote and cloud access, and per-shipment independent US lab testing. A roughly 25% charge on qualifying sales is often described as part of this regime; it is a Section 232 tariff announced separately by the White House in January 2026, not a condition inside the BIS rule.
China responded by discouraging or blocking imports at customs: Nvidia prepared roughly 82,000 H200s for the market and shipments stalled within about a day of US approval, with China-configured production halted in March 2026 and only modest deliveries resuming in August. Samsung and SK Hynix now operate their China fabs on annual licences rather than indefinite authorizations, sized for maintenance rather than expansion. Separately, the Remote Access Security Act — which would extend controls to cloud access, closing the obvious workaround — passed the House 369–22 in January 2026 and remains pending in the Senate. It is a bill, not law.
For the economics, export policy functions as a segmentation device on the global supply curve: price-discriminated regional markets, a structural bid for compliant offshore capacity, a compliance premium in cross-border compute contracts, and a policy-risk overlay that any compute forward implicitly prices. What happens to a compute contract when the underlying hardware’s export status changes is exactly the sort of event-risk clause commodity lawyers spent decades refining in oil and power.
Benchmark regulation arrives with the market. As compute indices become settlement references for cleared derivatives, they inherit the benchmark-integrity apparatus — IOSCO’s Principles and, for EU and UK use, the Benchmarks Regulation. The August 2026 move of Compute Desk’s benchmarks under GX Benchmarks, an FCA-regulated administrator, is the template: index origination by market specialists, administration by a regulated party. Transaction-based methodologies versus survey-based ones will face exactly the scrutiny the LIBOR episode taught the market to apply.
Prudential edges. As banks, insurers and clearinghouses take compute exposure, capital treatment, margin methodology and concentration rules for a brand-new, fast-depreciating, policy-sensitive collateral class all have to be invented. That work is barely begun, and it is where risk practitioners have the most to contribute.
Is there an explicit closed-form relationship between the physics above and the prices printing on a compute curve? No — and the reason is instructive. For a storable asset the forward is pinned by arbitrage: buy spot, store it, deliver against the forward. A GPU-hour cannot be stored. An hour not sold at three o’clock is gone, like a megawatt-hour or an airline seat. With no cash-and-carry there is no arbitrage mechanically tying forward to spot, so compute forwards behave like electricity forwards: an expectation plus a risk premium whose sign is set by which side needs the hedge more. In power, consumers historically paid producers for certainty. In compute it is arguably the reverse — capacity owners carrying debt service need price certainty more than tenants do, which biases forwards below expected spot.
But compute-hours cannot be stored while the machines that produce them can be built, redeployed and switched off. Three anchor relationships discipline where a rational forward can print.
The forward cannot sustainably print below the variable cash cost of the marginal operator — electricity, colo, minimal staffing. Below it, capacity withdraws rather than sells. Identical to the retirement condition on a power plant.
No rational buyer pays more for legacy hours at a future date than the same capability costs on the best chip deployable by then. If a new generation delivers its claimed cost-per-token improvement, that ratio is the ceiling collapsing onto every earlier SKU’s deferred months. If it slips, every legacy curve prints higher.
A rational inference operator pays at most what the output is worth: the forward is capped by the market price per token multiplied by the tokens per hour the chip can serve. Falling token prices drag this down; serving-stack gains partially offset. This is the arbitrage tying the compute curve to the token market.
For the marginal workload, rental-price ratios should track delivered-performance ratios. The H200/H100 spread is the cleanest example: same generation, but the H200 carries 141GB of HBM3e at 4.8 TB/s against the H100’s 80GB at 3.35 TB/s. For decode-heavy work that bandwidth advantage is worth a real premium; for compute-bound work it is worth little. The spread is therefore effectively a market price on the decode share of AI demand — a tradable expression of the workload shift in section 01. The B200/H100 spread prices a full generation gap and embeds both the performance multiple and availability. And the RTX 5090 is the outlier and a lesson in quality basis: a consumer card with no HBM and no high-speed interconnect, serving small-model inference and fine-tuning, trading on its own supply and demand with only weak coupling to the datacenter SKUs. In commodity terms it shares the family name but is a different grade — WTI versus a condensate.
Each curve should normally sit in backwardation, deferred months below prompt, because every passing month brings the substitution ceiling down. The informative prints are the violations. A legacy curve in contango says the market expects demand for older inference capacity to outrun the cascade, or expects the next generation’s supply to disappoint. That is a macro signal about the entire AI capex cycle available nowhere else — which is, ultimately, the strongest argument for these markets existing at all.
| # | Factor | Why it moves the curve |
|---|---|---|
| 1 | The substitution schedule | Next-generation delivery volumes and realized cost per token, plus the fab, HBM and packaging constraints that can delay them |
| 2 | Token demand growth and mix | Total inference demand and its decode intensity, which moves the SKU spreads specifically |
| 3 | Power and rack availability | A cascaded GPU without a powered socket is not supply — interconnection delays are bullish legacy rental rates |
| 4 | Cost-floor inputs | Electricity and colo pricing, which set the shutdown boundary |
| 5 | Contract and financing flow | Contract roll-offs hitting spot, distressed capacity sales, and vendor backstops absorbing supply — a backstop purchase is functionally a floor bid |
| 6 | Regulation | Export-policy shifts re-routing supply between regions; cloud-access rules; benchmark-governance changes affecting the index itself |
| 7 | Index basis | A transaction-based index moves with the composition of what prints — region, tenor, cluster size, SLA — independently of “the” price of an hour. A physical hedger carries this exactly as a crude seller carries basis to Brent |