Energy intensity of AI inference
Reasoning, batching, hardware and precision change the energy cost of a response.
Abstract. The programme synthesizes measured inference benchmarks into an environmental intensity framework that separates response-level energy, token efficiency, hardware choice and methodological uncertainty.
Moves beyond energy per token toward energy per accepted task.
The benchmark is large enough to expose regime changes.
Model families included in the reference benchmark synthesis.
Conversation, reasoning and multimodal workloads.
Hardware / batch / precision combinations.
H100 and B200 in matched comparisons.
Chung et al. (2026), Where Do the Joules Go? Diagnosing Inference Energy Consumption, ML.ENERGY, arXiv:2601.22076 ↗. The published measurement boundary and configuration-level results are retained in this research record.
For Qwen 3 32B, reasoning raised energy per response by ~23×.
At the reference settings, conversation energy was 95 J per response; reasoning was 2,192 J.
| Qwen 3 32B / B200 | Conversation | Reasoning |
|---|---|---|
| Mean output | 627 tokens | 7,035 tokens |
| Max batch | 512 | 128 |
| Energy / token @ BS128 | 0.209 J | 0.312 J |
| Energy / token @ max batch | 0.151 J | 0.312 J |
| Energy / response | 95 J | 2,192 J |
Optimization rules reverse across utilization regimes.
3–5×
Higher concurrency can cut energy per token by roughly three to five times in the reference analysis.
63 / 72
B200 won 63 of 72 matched-latency LLM comparisons, with a median ~35% energy reduction.
0 / 7
FP8 won none of seven small-batch comparisons; median energy was about 30% higher.
Production assumptions can move a model estimate by 65×.
A 400-token Mistral Small 3.2 estimate changed from 1.43 Wh to 0.022 Wh after the methodology incorporated more production-like batching and hardware assumptions.