Inference Eats 70% of Compute: In Autumn 2026, the Compute Industry Hits an “Efficiency Inflection Point”

Over the past week, the compute sector did not see any single isolated “explosive launch,” but something more symbolic took place: several originally parallel threads converged in the same time window into a clear turning signal—inference compute has comprehensively overtaken training in both spending and workload share, and the large-scale deployment of agents is pushing the compute industry from a “GPU-hoarding race” toward a “race for efficiency and returns.” This is not a simple industry slogan, but a result jointly indicated by policy data, demand structure, business ledgers, technical architecture, and capital doubts within the same week. The compute industry is still expanding rapidly, but the logic of expansion is changing: in the past, people asked “how many more cards can be bought”; now they ask “how much can one card earn back per hour.” In the past, people looked at cluster scale; now they look at effective output per unit of compute. In the past, compute was treated as capex; now more and more people treat it as a productive asset that must be tested by depreciation, electricity prices, utilization, and payback cycles.

I. One Inflection Point, Three Sets of Data

On October 20, the Ministry of Industry and Information Technology (MIIT) disclosed at a State Council Information Office press conference that China’s intelligent computing power had reached 2,592 EFLOPS by the end of September, up 146% year on year, and explicitly stated that the next step would be to “accelerate the expansion and quality improvement of computing facilities.” These figures themselves are a continuation of the supply side, but what truly defined the direction of the week was the structural change on the demand side. 2,592 EFLOPS shows that compute supply is still expanding rapidly, and the 146% year-on-year growth rate shows that intelligent compute remains a key direction of infrastructure investment. But the phrase “expand capacity and improve quality” placed together means that the policy goal is no longer merely to make the scale larger, but to improve utilization, network coordination, green energy efficiency, and actual output while expanding scale. In other words, the evaluation system for compute construction is shifting from “how much has been built” to “how well it is used.”

The China Academy of Information and Communications Technology (CAICT), in its “Token Economy Development Research Report (2026),” forecasts that in 2026 inference compute will account for 70.5% of total AI compute workload, and AI compute demand is accelerating its shift from model training to inference and agent operation. The significance of this ratio lies in the fact that it changes the form of compute demand. Training workloads are usually phased, centralized, and plannable: a large-model training task can consume a large amount of compute over a relatively defined period and then end. Inference workloads are closer to production systems: they are distributed across user requests, agent tasks, tool calls, long-context processing, and multi-turn iteration, featuring continuity, volatility, and a long-tail nature. When inference accounts for more than 70%, it means compute centers are no longer just “training factories,” but increasingly “Token factories” and “agent runtime foundations.”

Gartner’s statistical lens gives the other side of the same conclusion: in 2026 global inference spending will reach $23.3 billion, surpassing training spending of $19 billion for the first time. Changes in spending structure affect business decisions more directly than workload share. High training spending corresponds to the model-capability race and front-loaded capex; high inference spending corresponds to service revenue, call volume, unit costs, and gross margins. For cloud providers, compute leasing companies, and model service providers, this means the revenue model must be recalculated: it is no longer enough to build a cluster and wait for training customers to book it exclusively; instead, one must compete for sustained call demand in the inference market through throughput, latency, caching, scheduling, and pricing. Inference spending surpassing training spending does not mean training is unimportant; it means the value-realization link of the compute industry is shifting from “one-off training investment” to “continuous inference services.”

What drives this shift is agents moving from demonstrations to large-scale deployment. In his GTC Berlin speech on October 21, Jensen Huang re-deconstructed the ledger of compute demand: agents have the ability to think, reason, iterate, and use tools, and the number of Tokens consumed by a single task is amplified by multiples compared with traditional one-question-one-answer interaction. Traditional chat interaction is often one question and one answer, with relatively controllable Token consumption; an agent may first plan a task, then call tools, read results, correct errors and retry, and finally generate a summary. Behind a single user request may be multiple rounds of model calls, multiple context assemblies, and multiple external tool interactions. Token consumption is amplified, and inference compute demand is no longer linear growth; it may show a multiplier effect driven by task complexity. At the same time, agents have higher requirements for latency, stability, and concurrency, and compute scheduling cannot look only at peak compute, but also at the coordination efficiency of networks, storage, caching, and inference frameworks.

On the same day, Meta’s earnings provided an operating sample of consumer-grade agents—about a month and a half after Muse launched, more than 10 million users had completed configuration, shopping became the largest use case, compute consumption was significantly higher than that of any previous AI product, and Meta therefore proposed to improve compute efficiency tenfold within a year. The value of this sample lies in showing how consumer-grade agents push compute demand from “model R&D departments” to “production systems serving massive users.” The shopping scenario means recommendation, price comparison, decision assistance, multi-turn dialogue, and tool calls are superimposed; a seemingly simple user need may be backed by a complex reasoning chain. The larger the user scale, the more continuous the inference load, and the closer compute costs come to operating costs rather than R&D costs. Meta’s proposal to improve compute efficiency tenfold within a year also shows that even leading platforms cannot solve inference economics only by stacking cards; they must seek efficiency from model compression, inference frameworks, cache reuse, task scheduling, and hardware co-design.

The structural reversal on the demand side means the valuation anchor of the compute industry is shifting: in the past, people looked at “how many cards and how large the cluster”; now they look at “how many valuable Tokens one card can produce per hour.” This change will transmit to multiple links in the industrial chain. Chipmakers cannot talk only about peak compute, but also about inference throughput, energy efficiency, and interconnection capability. Cloud providers cannot talk only about data center scale, but also about utilization and unit Token cost. Model companies cannot talk only about parameters and leaderboards, but also about inference pricing and gross margins. Compute leasing companies cannot talk only about deployment rate, but also about customer structure and long-term lease stability. Compute is still a scarce resource, but the definition of scarcity is shifting from “whether there are cards” to “whether there is efficient, stable, billable effective compute.”

II. DeepSeek’s $38: A Verifiable Efficiency Sample

If inference surpassing training is the “quantity” inflection point, then the operating data disclosed by DeepSeek on October 20 when it open-sourced its inference framework DeepInfer attempts to answer the “quality” question. Based on this framework, DeepSeek’s single cluster achieved service throughput of 155 trillion Tokens per day, with overall revenue of $38 per card per hour, based on off-peak pricing. The reason this figure triggered industry resonance is not that it is “high,” but that it provides a previously rarely disclosed comparison: the list rental price of NVIDIA H200 per card per hour is about $6–8, and B300 about $15–20; even at actual market transaction prices, long-term B300 leases can be as low as $5–6 per hour. For the same card, there is a several-fold gap between the earning capability of direct rental and the earning capability of using it to sell Tokens.

The key to this comparison is not simply concluding that “selling Tokens is more profitable than renting cards,” but that it places inference compute before an auditable business ledger for the first time. In the card-rental model, revenue is rent, and costs are depreciation, electricity, network, data center, and O&M; in the Token-selling model, revenue is call fees, and costs include not only the above items but also model R&D amortization, inference framework efficiency, cache hit rate, scheduling loss, and pricing strategy. The reason $38 per card per hour matters is that it turns compute from “a number in the capex column” into a productive asset about which one can ask “how much it earns back per card per hour.” For compute centers, if direct card rental can only bring in a few dollars while efficient inference services can generate higher unit revenue, then the competitive focus will shift from “who owns the cards” to “who can make cards produce more billable Tokens.”

Overseas social media therefore estimated that DeepSeek’s inference business hardware gross margin may fall in the 84%–96% range. It must be emphasized that this estimate relies on multiple assumptions such as cache hit rate, throughput, and pricing, and is not a full profit basis; DeepSeek also did not disclose the chip models, quantities, and complete cost structure of its cluster. High gross margin estimates easily cause excitement but are also easily misunderstood. Hardware gross margin covers only part of the costs and does not include full costs such as amortization of model training investment, R&D expenses, marketing expenses, electricity fluctuations, network bandwidth, and data center construction. Off-peak pricing also means $38 is not an all-day average price, and actual revenue will fluctuate with the load curve. Therefore, this figure is better understood as an efficiency signal: under specific frameworks, specific loads, and specific pricing conditions, the unit output of inference compute can be significantly higher than that of a simple card-rental model. It proves that there is huge room for efficiency optimization, but it cannot directly infer that the entire industry can achieve the same profit margin.

Its significance also lies in the fact that competition in inference compute is beginning to shift from “hardware parameters” to “systems engineering.” The throughput of 155 trillion Tokens per day is not only the result of chip compute, but also involves the batching capability of the inference framework, KV cache management, speculative decoding, operator optimization, cluster scheduling, and network communication. After DeepInfer was open-sourced, the industry can discuss more concretely where inference efficiency comes from: Is it increasing batch size, reducing time to first token, improving cache hit rate, or optimizing multi-GPU parallel communication? In the past, discussions in the compute industry often stopped at chip models and cluster scale; DeepSeek’s sample pushes the question to “how many effective Tokens can the same card produce under different inference frameworks and scheduling strategies.” This happens to echo the policy side’s “expand capacity and improve quality.” The MIIT’s “15th Five-Year Plan for the Development of the Information and Communications Industry,” released in September, proposed that by 2030 intelligent computing power should grow more than fivefold compared with 2025, and placed “quality improvement” alongside “capacity expansion.” When inference becomes the main body of compute consumption, the economic meaning of “quality improvement” becomes concrete: not building more cards, but making existing cards produce more billable Tokens.

Looking further, improvements in inference efficiency will change the tightness of compute supply and demand. If unit compute can serve more users and handle more tasks, then short-term marginal demand for new compute may slow; but if efficiency improvements bring prices down, they will stimulate more agent applications to come online, which in turn will increase total compute consumption. This is a typical interaction between efficiency and demand: the higher the efficiency, the lower the unit cost, the more widespread the applications, and total compute demand may not decrease but may instead grow faster. What truly determines the health of the industry is not total compute scale alone, but whether efficiency improvements can be converted into sustainable revenue and profits. DeepSeek’s $38 provides a starting point, but it is more a question than an answer: can the entire industry replicate this efficiency under real workloads, full costs, and long-term pricing?

III. The Supply Side’s Response: From “Delivering Compute” to “Delivering Compute That Can Pay Back”

The flow of capital and technology also confirms the same direction. From October 20 to 21, Anthropic was reported to be negotiating a second compute leasing agreement of about $45 billion with Nscale; if completed, the two agreements would total about $90 billion and continue to use NVIDIA’s next-generation Vera Rubin chips. Such ultra-large-scale agreements show that leading model companies still need long-term, locked-in compute supply, and the combined demand for training and inference is far from over. But the form of the agreement itself is also worth noting: it is not simply buying cards, but leasing compute, locking capacity, and binding next-generation chips. What the compute supplier must deliver is not only hardware, but also deployment speed, network capability, power supply assurance, and long-term O&M. For the demand side, compute contracts are increasingly like long-term infrastructure contracts, which must both ensure supply and control unit costs.

Musk also gave SpaceX’s compute plan: 13 GW online by the end of next year, with an extreme case of 20 GW, cloud compute rent exceeding $50 billion per GW, and the payback period of compute assets being repriced. Gigawatt-level compute planning pushes the issue to the energy and infrastructure level. Compute is no longer just a chip issue, but a comprehensive issue of electricity, cooling, land, grid connection, and capex. When a single gigawatt of compute corresponds to tens of billions of dollars in rental scale, the payback period, utilization, and sensitivity to electricity prices of compute assets will become core variables. The further the compute race goes toward large scale, the less it is a pure chip-procurement race and the more it is a race in energy infrastructure and capital structure.

At the technical architecture level, the domestic 100,000-card ultra-large cluster and Huawei Ascend 1,024-card liquid-cooled supernode exhibited at WEC2026 in Shenzhen this week, as well as NVIDIA Vera Rubin’s fully liquid-cooled 72-GPU rack design, jointly point to a trend: the competitive unit is upgrading from “single-card performance” to “whole-rack, supernode, and cluster-level coordination.” Single-card performance is still important, but inference and agent workloads depend more on cluster efficiency. Multi-card interconnection, liquid-cooling heat dissipation, power density, failure recovery, and scheduling systems determine whether large-scale clusters can operate stably. The 100,000-card ultra-large cluster and the thousand-card-level liquid-cooled supernode show that domestic compute is shifting from catch-up at the single-chip level to system-level integration; the fully liquid-cooled rack shows that high-density compute must be supported by more energy-efficient cooling solutions. Future compute competition is not only about whose chip peak is higher, but also about whose cluster is more stable, more power-efficient, easier to operate and maintain, and better able to convert compute into effective Tokens under real workloads.

These actions on the supply side jointly point to “delivering compute that can pay back.” In the past, the common logic of compute projects was “build first, find customers later”; now capex is too large and payback cycles too long, so demand must be locked in advance, energy efficiency optimized, and utilization improved. Compute leasing companies need to pay more attention to customer structure: whether customers are training or inference customers, short-term or long-term leases, whole-cluster bookings or Token-based billing. Cloud providers need to pay more attention to the gross margin of inference services rather than simply expanding GPU deployment rates. Chipmakers need to pay more attention to inference throughput and energy efficiency, because the higher the share of inference workloads, the more important the electricity cost per Token becomes. The policy side’s “expand capacity and improve quality” and the market side’s “payback” intersect here: compute infrastructure still needs to expand, but expansion must match demand realization, energy conditions, and commercial returns.

IV. The Other Side of Efficiency: Depreciation, Marginal Pricing, and the Return Gap

But this narrative of “compute assets can pay back” also encountered serious doubts from the other end this week. On October 20, Gartner reiterated a judgment that unsettles the market: up to $3.5 trillion in AI infrastructure investment may be at risk, because between 2027 and 2030 overall AI revenue will not be enough for planned capex to earn a profitable return. Its chief forecaster said bluntly that “there is currently more compute capacity than demand, and large amounts of Tokens and processing capacity are being given away through freemium offerings.” The sharpness of this judgment is that it does not say AI has no demand, but that the speed and price level of demand realization may not be enough to cover massive capex. Compute supply can be deployed quickly within one or two years, but enterprise adoption, agent diffusion, and the formation of payment habits take longer. If supply grows faster than revenue, the compute market may experience阶段性 overcapacity.

Citadel founder Ken Griffin, on October 21, broke the potential trap of compute investment into three layers: first, the marginal operating cost of GPUs is extremely low, and once supply exceeds demand, spot rental and API prices may “free fall,” repeating the trajectory of dark fiber in the 2000 telecom bubble; second, the situation of charging high rents with 5–6-year-old GPUs is likely unsustainable, and the 7–9-year depreciation model faces systematic revaluation; third, enterprise AI spending has shifted from trial and error to rigorous ROI calculation, and “save where possible” will form direct tension with giants’ massive capex. These three judgments correspond to price, depreciation, and demand, respectively, and precisely hit the most vulnerable parts of compute assets.

The first layer is marginal pricing. Once a GPU cluster is built, marginal operating costs are mainly electricity, O&M, and network, which are not high relative to construction costs. If the market is oversupplied, compute leasing companies may rent at prices close to marginal cost in order to amortize depreciation, causing spot prices to fall rapidly. This is similar to dark fiber in the telecom bubble: once fiber is built, the marginal cost of use is very low, but demand does not materialize in sync, and price wars destroy investment returns. The compute market may not repeat this exactly, but inference API price wars have already shown the pressure of marginal pricing. When model service providers compete for call volume by cutting prices, unit Token revenue falls, and the payback period of compute assets is lengthened.

The second layer is depreciation. GPUs are not permanent assets; technology iterates quickly, and energy efficiency and performance improve every year. Whether a 7–9-year depreciation model is reasonable depends on whether old GPUs can still obtain sufficient rent or inference revenue a few years later. If new-generation chips are significantly better than old cards in unit Token cost, the rent of old cards will fall, and the depreciation period may be overestimated. For compute leasing companies, the most dangerous thing is not having no customers, but customers being willing to pay increasingly lower prices only for old cards while book depreciation has not yet completed. For cloud providers, continuously purchasing new cards brings capex pressure, but if they do not upgrade, unit compute cost will lag behind. Revaluation of the depreciation model will directly affect the valuation and financing capacity of compute assets.

The third layer is enterprise ROI calculation. Enterprise AI spending has shifted from “give it a try” to “do the math.” When enterprises find that certain AI functions cannot bring clear revenue or cost savings, they will cut call volume or demand lower prices. This forms direct tension with giants’ massive capex: the supply side is expanding, while the demand side is doing rigorous calculations. Although agents amplify Token consumption, enterprises will not pay unconditionally for all Tokens; only tasks that bring actual value will continue. This means inference compute demand growth will be structurally divergent: compute demand in high-value scenarios is strong, while low-value scenarios may be compressed or made free. Gartner’s statement that “large amounts of Tokens and processing capacity are being given away through freemium offerings” is exactly a manifestation of this divergence.

These warnings do not deny the long-term growth of compute demand, but they point to a timing mismatch: the depreciation speed of compute assets may be faster than the speed at which demand is realized. Another piece of news this week provided a physical footnote—Amazon has nearly $5 billion worth of GPUs idle due to power shortages. When power, cooling, and grid connection progress cannot keep up with chip deployment, “capacity expansion” itself does not automatically equal “quality improvement.” GPUs are bought and data centers are built, but if there is not enough power, grid connection is slow, and cooling capacity is insufficient, compute cannot be converted into revenue. This shows that the bottleneck of compute infrastructure is not only chips, but also energy and engineering capability. For compute projects, power supply assurance and grid connection progress may determine actual deployment rate and revenue realization more than chip procurement.

V. Industrial Chain Impact: Valuation Anchors, Pricing Power, and Indicators to Watch

Looking at this week’s threads together, the valuation anchor of the compute industrial chain is undergoing a three-layer shift.

The first layer is from “compute scale” to “effective compute.” Total compute scale is still important, but investors and customers will pay more attention to deployment rate, utilization, inference throughput, and unit Token cost. A 100,000-card cluster with insufficient power, network congestion, and low scheduling efficiency may have actual effective compute far below its nominal scale. Conversely, a smaller cluster with efficient scheduling, high cache hit rate, and an optimized inference framework may generate higher unit revenue. Effective compute becomes a more real competitiveness indicator.

The second layer is from “card rental revenue” to “Token revenue.” The revenue of the card-rental model is relatively stable, but its ceiling is constrained by rental prices; the Token revenue model is more elastic but affected by call volume, pricing, and competition. If compute leasing companies provide only bare metal, they may fall into price wars; if they can provide inference services, model hosting, and agent runtime environments, they may obtain higher added value. Competition among cloud providers will also shift from GPU deployment speed to inference platform capabilities, including batching, caching, monitoring, elastic scaling, and billing models.

The third layer is from “capex” to “payback cycle.” Compute projects are no longer just financing and construction issues, but operation and return issues. The larger the capex, the longer the payback period, and the more sensitive to utilization and electricity prices. Financing costs, depreciation policies, power contracts, and long-term customer contracts of compute assets all affect project feasibility. If the policy side’s “expand capacity and improve quality” can promote coordination among green power, liquid cooling, networks, and scheduling, it will help shorten the payback period; if it only expands scale, it may exacerbate supply-demand mismatch.

For observers, several sets of indicators can be watched in the coming quarters: whether the share of inference spending continues to rise; whether Token consumption brought by agent applications is converted into sustained payment; whether spot rental prices and API prices for compute fall rapidly; whether GPU depreciation periods and secondary market prices are revalued; whether data center deployment rate, utilization, and power supply assurance improve; the throughput and energy efficiency of domestic supernodes and liquid-cooled clusters under real workloads; and whether the inference gross margins of leading model companies can hold up under a full-cost basis. These indicators reflect industry health better than compute scale alone.

VI. In the Second Half of the Compute Race, It Is Not About Who Is Bigger

Looking back at the past week, the hottest topic in the compute sector was not a single product, a single order, or a single policy, but a new consensus taking shape: inference compute has become the main body of demand, agents are pushing compute consumption from intermittent training to continuous, long-tail production workloads, and the commercial rationality of compute assets is for the first time being required to prove itself through “return per card per hour.”

This does not mean training is unimportant, nor does it mean capacity expansion will stop. The scale of 2,592 EFLOPS and the goal of fivefold growth in the “15th Five-Year Plan” determine that the supply side will still expand rapidly. Training still requires large-scale clusters, the frontier model race still requires top-tier compute, and domestic compute system-level integration still requires continuous investment. But the focus of competition is shifting: in the first half, the competition was about who could get more cards and build larger clusters; in the second half, the competition is about who can use the same cards to produce more billable effective Tokens, and whose inference efficiency can make revenue per unit of compute outrun depreciation, electricity, and capital costs.

DeepSeek’s $38 provides a starting point, but it is more a question than an answer. When Gartner’s $3.5 trillion warning and the MIIT’s capacity expansion plan appear in the same week, when Vera Rubin’s rack density and Amazon’s idle GPUs become news at the same time, the real question the compute industry needs to answer is no longer “how much more compute is needed,” but “at what efficiency, at what time, and for whom will this compute generate returns.” The inflection point has arrived, and the whistle for the efficiency race has just blown. For chipmakers, cloud providers, model companies, compute leasing companies, and investors, the most important capability going forward may not be to continue amplifying compute scale, but to turn compute into effective productivity that can be measured, priced, and paid back.