Watts to Tokens — how electricity becomes intelligence

August 7, 2026

Every industry has a signature conversion. In metallurgy, which I studied at IIT Kharagpur and promptly forgot, the signature conversion is the blast furnace: iron ore, coke and limestone go in at the top, hot metal comes out at the bottom, and the entire economics of a steel plant hangs on one ratio called the coke rate - kilograms of coke burnt per tonne of metal produced. Reduce the coke rate and you win. That is the whole game.

A decade later I find myself in the AI infrastructure business, and the same plant is staring back at me. Feedstock in: electricity. Product out: tokens - the little chunks of text that models read and write. Everything in between - the GPUs, the exotic networking, the buildings, the cooling loops, the billion-dollar capex announcements you keep reading about - is the furnace - popularly known as the "AI Factory" nowadays. And the ratio everyone is quietly optimising is the modern coke rate: watts per token.

At NAVA, we build AI compute clusters, which means I sit at the watts end of this pipeline. This essay is the 101 I wish someone had handed me when I started. No prerequisites beyond engineering common sense.

One caveat - since it is a beginners' guide, written by a beginner - there may be inaccuracies, which I am sure you will excuse me for.

The product: tokens

Start from the output. A token is roughly three-quarters of an English word. Common words like "the" are one token; "infrastructure" is one token because it appears so often; "Sambuddha" is probably three or four because it does not. When you send a prompt to ChatGPT or Claude, your text is chopped into tokens, and the reply comes back one token at a time. That stream of tokens is the product. It is what API providers bill you for, per million.

A large language model is, mechanically, a next-token predictor. Given everything so far, it outputs a probability distribution over what comes next, one token is sampled, appended, and the loop runs again. All the apparent reasoning, coding and poetry is this loop running very fast.

The model itself is just numbers - parameters, also called weights. A frontier model today carries hundreds of billions to a few trillion of them, arranged as large matrices. When a model "thinks", what physically happens is matrix multiplication: your input, multiplied against those weight matrices, layer after layer, until the distribution over the next token falls out. That is the whole trick. The rest of this essay is about the machinery required to do that multiplication at civilisational scale.

The machine: why GPUs and not CPUs

A CPU is a few very clever cores, built to do complicated things one after another. A GPU is thousands of simple cores, built to do the same dumb thing to a lot of numbers at once. Matrix multiplication is exactly that kind of dumb thing. Think of it as the difference between one professor solving a hard problem versus the entire first-year batch doing arithmetic drills simultaneously. For matrix multiplications, you want the first-years, all four thousand of them.

Two numbers matter when you look at a GPU spec sheet. First, FLOPs - floating point operations per second, the raw arithmetic rate. An NVIDIA H100, the workhorse chip of the 2023-24 buildout, does about a petaflop of 16-bit maths. Second, and less obviously, memory bandwidth. The weights live in special stacked memory called HBM (high bandwidth memory) sitting millimetres from the compute die - about 80 GB of it on an H100, readable at about 3.35 TB/s. Here is the industry's dirty secret: a modern GPU can do arithmetic far faster than it can fetch the numbers to do arithmetic on. Much of AI systems engineering is really the art of feeding the beast.

The third thing to understand is that a single GPU stopped being the unit of compute years ago. Inside a rack, GPUs are wired together with NVLink, a fat interconnect that lets 72 chips behave like one enormous GPU - that is what NVIDIA's GB200 NVL72 rack is. Across racks, clusters are stitched together with InfiniBand or high-end Ethernet. The unit of compute today is the rack.

Training: the expensive childhood

Training is how the weights get their values. Pre-training, the big first phase, is brutally simple in concept: take a giant pile of text - on the order of 15 trillion tokens for a recent frontier model, which is most of the useful internet - and play "predict the next token" over it, again and again. The model guesses, the guess is scored against the actual next token, and an algorithm called backpropagation works out which of the trillion knobs to nudge and in which direction. Then all of them are nudged, a little. Repeat a few crore times over some months, and out of that comes something that can draft your appraisal email.

The interesting part is what this demands physically. Frontier training runs use tens of thousands of GPUs, and here is the catch - they must run as one synchronised machine. It is a group project where nobody may write line two until everyone has shared line one. This is why AI clusters use networking that looks nothing like your office LAN, and why one slow link or one flaky chip drags down the whole convoy.

All of this is why training costs what it does. GPT-4's training bill was rumoured north of $100 million, and that was 2023 money; one frontier lab has estimated that a single training run will want around five gigawatts of power by 2027, which is the output of several large power plants pointed at one model.

Pre-training is followed by post-training, which is shorter but decides the personality: supervised fine-tuning on curated examples, then reinforcement learning from human feedback (RLHF) to make the model helpful rather than merely knowledgeable, and lately, reinforcement learning on maths and code to teach models to reason step by step. Think of pre-training as the degree and post-training as the induction week, except the induction week is where the discipline gets installed.

Inference: the long adult life

Training happens once. Inference - actually serving the model - happens every time anyone sends a message, forever after. By most estimates the large majority of AI compute now goes to inference, not training. Training is the R&D capex; inference is the factory running three shifts.

Every query is a forward pass through all the weights, and it has two very different phases. Prefill is the model reading your prompt - all your input tokens can be processed in parallel, so this phase is compute-bound and fast. Decode is the model writing its answer, and it is the awkward part: tokens come out strictly one at a time, and for every single token, the hardware must stream essentially all the weights through the chip again. Reading the question paper in one glance, then writing the answer one word at a time while re-reading the entire syllabus before each word. This is why decode is limited by memory bandwidth, not arithmetic, and why that 3.35 TB/s number matters more than the petaflop.

To avoid recomputing the conversation so far on every step, the model keeps a running scratchpad called the KV cache - its working memory of your chat. The KV cache grows with the length of the conversation, which is exactly why long context windows are expensive: a million-token context is not a marketing problem, it is a memory-capacity problem measured in gigabytes per user.

The economics of serving come from batching. A GPU answering one user at a time is a mess counter serving one thali per pass - criminal waste. Stack fifty users' requests into one pass and the same weight-streaming cost is shared across all of them. Inference providers live and die by how cleverly they batch without wrecking latency.

Two recent phenomena sit on top of this. First, reasoning models: the newer systems that "think" before answering are spending extra tokens at inference time - sometimes ten to a hundred times more compute per question - to buy quality. It means demand for inference is growing even faster than user counts. Second, prices: for a fixed level of capability, the price per token has been collapsing roughly ten-fold a year, through better chips, better software and smaller models matching bigger ones. You would think cheaper tokens mean less total spend. The opposite has happened - usage grows faster than prices fall, so total consumption keeps climbing. James Watt's steam engine made coal use more efficient and coal consumption exploded; fitting that the man's name is also the unit this essay is about.

The physical layer: where the watts go

Now to my side of the plant. A GPU is, thermally speaking, a very expensive room heater. An H100 draws about 700 watts; the Blackwell generation runs 1,000-1,400 watts per chip. Stack 72 of them into a GB200 NVL72 rack with CPUs, switches and power electronics and the rack draws 120-130 kilowatts. For calibration, a normal cloud-computing rack of the last decade drew 5-10 kW, and the previous Hopper-generation AI rack about 40. The Vera Rubin racks entering volume in late 2026 sit around 200 kW, the 2027 "Rubin Ultra" generation is specified at roughly 600 kW per rack, and megawatt-class racks are on public roadmaps. The IEA has pointed out that a single modern AI rack draws what 60-odd households do. One rack, one apartment block, same electricity bill.

Physics is unsentimental about where those watts go: every one of them becomes heat, in a box the size of a fridge, and the heat must leave or the silicon throttles. Air, our default coolant since the beginning of computing, gives up somewhere in the 30-50 kW per rack range - you simply cannot move air fast enough through that volume without the data hall becoming a wind tunnel. Water carries roughly 3,000 times more heat than the same volume of air, so the industry has gone back to plumbing. The mainstream answer is direct-to-chip liquid cooling: a cold plate sits on each GPU, coolant loops through it, and a unit called a CDU (coolant distribution unit) exchanges that heat into the facility's water system. The pleasant surprise is that the coolant does not need to be cold - these loops run happily at 30-45°C, warm enough that in many climates you can reject the heat outdoors through a dry cooler with no chiller at all, which is a large efficiency win. For the truly dense stuff there is also immersion cooling, where entire servers are dunked in tanks of dielectric fluid, which sounds like science fiction and smells like mineral oil.

The catch is that almost every data centre on Earth was designed for air. Retrofitting an air-era facility for liquid - new pipework, manifolds, CDUs, floor loading, and increasingly the new 800-volt DC power distribution the latest racks want - is a serious engineering programme.

One metric ties the facility together: PUE, power usage effectiveness - total facility power divided by power that reaches the IT gear. A legacy air-cooled site runs 1.5-1.8, meaning 50-80% overhead on cooling and losses; a well-built liquid-cooled facility runs 1.1-1.3. The overhead is not small change: cooling alone is forecast to consume about 195 TWh globally in 2026, which is more electricity than most countries use in total.

Watts are the bottleneck now

Zoom all the way out and the numbers get properly large. Gartner's mid-2026 forecast puts global data centre electricity consumption at about 565 TWh this year, up from 447 in 2025 - call it a third of India's entire annual electricity consumption, spent on server halls. In capacity terms that is about 132 gigawatts of demand in 2026, heading towards 290 GW by 2030; the IEA expects consumption to nearly double to around 950 TWh by 2030, roughly 3% of all electricity on the planet.

Here is the asymmetry that defines the moment: chips arrive in months, power arrives in years. A GPU order is a purchase order; a grid connection is an interconnection queue, a substation, transmission lines and permits, and that pipeline runs three to seven years in most countries. So the frontier players have stopped waiting - gas turbines installed behind the meter, long-term nuclear power purchase agreements, small modular reactor bets, and gigawatt-scale campuses planned next to generation (the Stargate programme alone targets 10 GW, with roughly half a trillion dollars attached). Two years ago the scarce resource was GPUs. Today it is powered land. The industry's competitive question has quietly become the plant manager's question: tokens per watt, and where do the watts come from.

The view from India

India is already one of the largest consumers of these models - by user count we are near the top of every AI app's dashboard. But nearly all of those tokens are generated on machines sitting in the United States. Our entire installed data centre base is on the order of one and a half to two gigawatts - less than what a single large American AI campus intends to draw by itself. That gap is the opportunity, and it is why capital is pouring into Mumbai, Chennai and Hyderabad, why "sovereign AI" and data-residency rules keep showing up in policy papers, and frankly why companies like the one I work at exist.

India's hand is interesting. On the plus side: enormous domestic demand, cheap and rapidly growing renewable generation, and engineering talent. On the minus side: a grid with its own queue problems, and a climate where the outside air in May is not exactly a cooling asset. The saving grace is that the liquid-cooling shift lands in our favour - warm-water direct-to-chip cooling cares far less about ambient temperature than air cooling ever did. A 40°C afternoon in Chennai is a much smaller penalty for a 45°C water loop than it was for a hall full of fans. The countries that figure out powered, plumbed, permitted land fastest will host the next decade's furnaces. There is no law of nature that says they must all be in Virginia.

Closing the loop

So the full pipeline, end to end: sunlight or coal or split atoms become grid electricity, which steps down through substations into an 800-volt busbar, into a rack, into a GPU, where it becomes matrix multiplication, which becomes a probability distribution, which becomes a token, which becomes the next word of the email you asked the model to write - while every last watt, having done its brief work as arithmetic, leaves the building as warm water. Watts to tokens, with heat as the by-product and thermodynamics keeping the books, exactly like every furnace before it.

Somewhere in a KGP classroom in 2010, a thermodynamics professor was trying to make us care about Q. It has taken fifteen years, but I finally do - the most important industry of our time is, underneath everything, a Q-management problem with a very good marketing department. The blast furnace people spent two centuries grinding down the coke rate. The AI people have started on watts per token. Same plant, better product.

If you want to go deeper

Andrej Karpathy's Intro to Large Language Models and the rest of his Zero to Hero series - the best free education on how models actually work.
How to Scale Your Model - a sharp, readable book on the systems side of training and inference, from the Google DeepMind folks.
SemiAnalysis - the reference newsletter for the chips, racks and data centre side of the industry.
Epoch AI - careful public data on training compute, costs and trends.
The IEA's Energy and AI report - the definitive picture of the electricity side.