What Would It Cost to Rebuild Nemotron? I Did the Math
In March, someone opened an issue on NVIDIA's Nemotron repository with a simple question: how long did pretraining take? How many GPUs, for how many hours? Eight months later the issue is still open, with no replies.1
I wanted that number too, so I worked it out. I've written twice this month about how open Nemotron is, once about the model and once about its future, and both times I landed in the same place: you can download and run it, but you can't really rebuild it. This post is about the bill behind that sentence.
The back-of-the-envelope
There's a well-worn rule of thumb for training compute: roughly six calculations per parameter for every token the model reads. For a mixture-of-experts model like Nemotron, you count only the parameters that are active for each token.
Nemotron 3 Super, the base underneath the current 3.5 Super release, has 12 billion active parameters out of 120 billion. NVIDIA says it was pretrained on 25 trillion tokens.2 Six times 12 billion times 25 trillion is 1.8 × 10²⁴ calculations. That number means nothing to most people, me included, so it has to be turned into GPU hours and then into dollars.
NVIDIA doesn't publish GPU hours. DeepSeek did. Its V3 report says pretraining a model with 37 billion active parameters on 14.8 trillion tokens took 2.66 million H800 GPU hours, which it priced at $2 an hour.3 Run the same formula over DeepSeek's numbers and you get how much useful work it got from each GPU hour. NVIDIA trains on its newer Blackwell chips, using a 4-bit format built for speed, so I assumed each B200 does two to four times the work of DeepSeek's H800s. Rental listings this month put a B200 at roughly $4 to $8 an hour.4
Put that together and pretraining Nemotron 3 Super comes to roughly 365,000 to 730,000 B200 hours. At rental prices that's about $1.5 million to $6 million, with a middle estimate near $3 million. And that assumes one run that works the first time.
The run is only the visible part
Nemotron 3.5 Super wasn't trained from scratch. NVIDIA started from Nemotron 3 Super and ran about 1.3 trillion more tokens through it, mostly a trillion tokens of distillation, plus Indian-language, long-context and software-engineering stages.5 On its own, that's a few hundred thousand dollars.
The catch is the distillation. Its teacher was Nemotron 3 Ultra, a 550-billion-parameter model with 55 billion active, pretrained on 20 trillion tokens.6 Using the same arithmetic, Ultra's pretraining alone is around $11 million, somewhere between $5 million and $21 million. So the training chain behind one 120B model comes to somewhere between $7 million and $28 million, about $14 million in the middle.
To rebuild Nemotron 3.5 Super faithfully, you'd have to rebuild its teacher first.
Then there's the data. NVIDIA's report does include one real cost figure. Translating English documents into Indian languages for a synthetic corpus cost 0.335 GPU hours per million input tokens, after optimizations cut it from 0.835.5 The twelve main languages took about 1.5 trillion input tokens, which works out to roughly 500,000 GPU hours for translation alone, before quality checks and rewrites. Only a small slice of that corpus went into Super, yet the GPU hours for that one data job sit in the same range as my estimate for Super's entire pretraining run.
None of this counts the work that never ships. DeepSeek's famous $5.6 million figure came with a sentence most people skipped: it excluded "prior research and ablation experiments."3 Epoch AI looked at OpenAI, MiniMax and Z.ai, where it could get the numbers, and found that final training runs were only 10 to 23 percent of their research compute.7 I don't know NVIDIA's ratio. If it looks anything like theirs, the full cost is several times my estimate. Reinforcement learning, which NVIDIA describes running across thousands of GPUs, isn't in my chart at all.2
What opening the process would mean
NVIDIA has already opened a lot of the process: recipes, much of the data, and the reports that made this math possible. What can't be handed over in a pull request is the run itself. Open training, in the full sense, means someone outside NVIDIA can afford to repeat a $14 million chain of computation, plus whatever it cost to find out which run was worth doing. Not many universities, nonprofits or open-source foundations can write that check. Publishing every last setting wouldn't change the size of the GPU bill.
That's worth saying plainly, because "why don't they open the training?" sounds like a question about willingness. A large part of it is a question about money. You can copy a recipe. Copying a few million GPU hours is a different thing.
So, is Nemotron's training open? The instructions are, more than most. The openness index can't score the family as a whole, and my provisional reproducibility score for Lightning was a 1 out of 3. Being able to follow those instructions still depends on who can pay for the GPUs.
Research-based estimate using NVIDIA's model cards and technical reports, DeepSeek-V3's published training costs, Epoch AI's compute analysis and November 2026 GPU rental listings, checked November 18, 2026. These are order-of-magnitude estimates, not NVIDIA disclosures.
