Parametric Scaling Laws and Inference-Aware Compute Allocation
Abstract. A model that will serve many tokens should be smaller, and trained on far more data, than the compute-optimal rule for training alone prescribes. We fit the parametric loss surface of Hoffmann et al. to 90 IsoFLOP training runs at 6 compute budgets from $$10^{18}$$ to $$10^{22}$$ FLOPs, derive the training-optimal allocation in closed form, and then minimize total lifecycle compute when the model serves $$I$$ inference tokens. At $$C_{\text{train}} = 10^{23}$$ FLOPs and $$I = 100\text{T}$$, a 3.6B model trained on 14.4T tokens reaches the validation loss of the training-optimal 14.8B model (2.005 nats) with 66.4% less compute over its life. Set your own budget and serving volume on the Stage.
1. The loss surface
For a dense autoregressive transformer with $$N$$ non-embedding parameters trained for one epoch on $$D$$ tokens, we model validation cross-entropy (nats per token) as an irreducible floor plus two power-law penalties, one for finite capacity and one for finite datap. 7:
$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1} $$
A forward pass costs $$2N$$ FLOPs a token and a backward pass $$4N$$, so pre-training compute is
$$ C_{\text{train}}(N, D) = 6 N D \tag{2} $$
We estimate $$\theta = (E, A, B, \alpha, \beta)$$ by bounded non-linear least squares over the $$M =$$ 90 runs:
$$ \hat{\theta} = \underset{\theta}{\arg\min} \sum_{i=1}^{M} \left( L_i - E - \frac{A}{N_i^{\alpha}} - \frac{B}{D_i^{\beta}} \right)^{2} \tag{3} $$
The fit recovers
$$ \hat{L}(N, D) = 1.692 + \frac{401.9}{N^{0.339}} + \frac{422.5}{D^{0.281}} \tag{4} $$
with $$E =$$ 1.692, $$A =$$ 401.9, $$B =$$ 422.5, $$\alpha =$$ 0.339, and $$\beta =$$ 0.281. It explains the sweep almost entirely ($$R^2 = 0.99996$$); the Stage lists each constant's standard error.
2. Training-optimal allocation
Minimizing (1) subject to $$6ND = C_{\text{train}}$$ with multiplier $$\lambda$$:
$$ \mathcal{L}(N, D, \lambda) = E + A N^{-\alpha} + B D^{-\beta} + \lambda \left( 6 N D - C_{\text{train}} \right) \tag{5} $$
The first-order conditions are
$$ \alpha A N^{-\alpha} = 6 \lambda N D, \qquad \beta B D^{-\beta} = 6 \lambda N D \tag{6} $$
and eliminating $$\lambda$$ gives the condition along the training-optimal path:
$$ \alpha A \left(N^\right)^{-\alpha} = \beta B \left(D^\right)^{-\beta} \tag{7} $$
Substituting $$D^* = C_{\text{train}} / (6 N^*)$$ yields the power laws
$$ N^(C) = G \left( \frac{C}{6} \right)^{a}, \qquad D^(C) = G^{-1} \left( \frac{C}{6} \right)^{b} \tag{8} $$
$$ a = \frac{\beta}{\alpha + \beta}, \qquad b = \frac{\alpha}{\alpha + \beta}, \qquad G = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}} \tag{9} $$
At the fitted constants, $$a =$$ 0.453 and $$b =$$ 0.547: tokens should grow slightly faster than parameters. At $$10^{23}$$ FLOPs the training-optimal model has 14.8B parameters and 1.12T tokens (75.8 tokens per parameter) and reaches 2.005 nats, against 2.103 nats under Kaplan et al.'s parameter-heavy allocation.
3. Counting inference
Training-optimality ignores the cost of serving. Decoding costs $$2N$$ FLOPs a token, so a model that serves $$I$$ tokens over its life costs
$$ C_{\text{total}}(N, D; I) = 6 N D + 2 N I \tag{10} $$
To reach a target loss at least total cost, invert (1) for the tokens a model of size $$N$$ needs,
$$ D(N; L_{\text{target}}) = \left( \frac{B}{L_{\text{target}} - E - A N^{-\alpha}} \right)^{1/\beta} \tag{11} $$
and minimize the one-variable objective over $$N > \left( A / (L_{\text{target}} - E) \right)^{1/\alpha}$$:
$$ C_{\text{total}}(N) = 6 N , D(N; L_{\text{target}}) + 2 N I \tag{12} $$
Setting $$\mathrm{d}C_{\text{total}} / \mathrm{d}N = 0$$ gives
$$ \alpha A \left(N^\right)^{-\alpha} - \beta B \left(D^\right)^{-\beta} = \frac{\beta B \left(D^*\right)^{-\beta - 1}}{3} , I \tag{13} $$
For $$I > 0$$ the right side is positive, so the balance (7) tips: the capacity penalty is left larger than the data penalty. The lifecycle-optimal model is smaller than the training-optimal one and over-trained, at a much higher token-to-parameter ratio.
4. Results across budgets
Table 1 sets the two allocations side by side for $$I = 100\text{T}$$ tokens served, each budget's training-optimal loss as the target.
Table 1: Training-optimal and lifecycle-optimal allocations at 100T tokens served.
| Budget $$C_{\text{train}}$$ | Target loss | Training-optimal $$N^$$ / $$D^$$ | Lifecycle-optimal $$N^$$ / $$D^$$ | Tokens per parameter | Lifecycle compute saved |
|---|---|---|---|---|---|
| $$10^{20}$$ | 2.600 | 647M / 25.8B | 94M / 4.58T | 48,810 | 83.5% |
| $$10^{21}$$ | 2.329 | 1.8B / 90.7B | 303M / 6.49T | 21,426 | 80.3% |
| $$10^{22}$$ | 2.139 | 5.2B / 319B | 1.0B / 9.47T | 9,299 | 75.2% |
| $$10^{23}$$ | 2.005 | 14.8B / 1.12T | 3.6B / 14.4T | 4,005 | 66.4% |
| $$10^{24}$$ | 1.912 | 42.1B / 3.96T | 13.4B / 23.3T | 1,738 | 51.6% |
The saving falls as the budget grows, because a fixed 100T tokens of serving weighs less against a larger training run.
Conclusions
- Data grows a little faster than size. Across the 6 budgets, $$a =$$ 0.453 and $$b =$$ 0.547, close to the equal split Hoffmann et al. report.
- Serving changes the answer. At $$10^{23}$$ FLOPs and 100T tokens served, a 3.6B model at 4,005 tokens per parameter cuts lifecycle compute by 66.4%.
- Plan a model by its life. Every figure here can be recomputed for another budget or serving volume on the Stage, and traced to its script in the Model tab.