Parametric Scaling Laws and Inference-Aware Compute Allocation

Abstract. A model that will serve many tokens should be smaller, and trained on far more data, than the compute-optimal rule for training alone prescribes. We fit the parametric loss surface of Hoffmann et al. to 90 IsoFLOP training runs at 6 compute budgets from $$10^{18}$$ to $$10^{22}$$ FLOPs, derive the training-optimal allocation in closed form, and then minimize total lifecycle compute when the model serves $$I$$ inference tokens. At $$C_{\text{train}} = 10^{23}$$ FLOPs and $$I = 100\text{T}$$, a 3.6B model trained on 14.4T tokens reaches the validation loss of the training-optimal 14.8B model (2.005 nats) with 66.4% less compute over its life. Set your own budget and serving volume on the Stage.


1. The loss surface

For a dense autoregressive transformer with $$N$$ non-embedding parameters trained for one epoch on $$D$$ tokens, we model validation cross-entropy (nats per token) as an irreducible floor plus two power-law penalties, one for finite capacity and one for finite datap. 7:

$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1} $$

A forward pass costs $$2N$$ FLOPs a token and a backward pass $$4N$$, so pre-training compute is

$$ C_{\text{train}}(N, D) = 6 N D \tag{2} $$

We estimate $$\theta = (E, A, B, \alpha, \beta)$$ by bounded non-linear least squares over the $$M =$$ 90 runs:

$$ \hat{\theta} = \underset{\theta}{\arg\min} \sum_{i=1}^{M} \left( L_i - E - \frac{A}{N_i^{\alpha}} - \frac{B}{D_i^{\beta}} \right)^{2} \tag{3} $$

The fit recovers

$$ \hat{L}(N, D) = 1.692 + \frac{401.9}{N^{0.339}} + \frac{422.5}{D^{0.281}} \tag{4} $$

with $$E =$$ 1.692, $$A =$$ 401.9, $$B =$$ 422.5, $$\alpha =$$ 0.339, and $$\beta =$$ 0.281. It explains the sweep almost entirely ($$R^2 = 0.99996$$); the Stage lists each constant's standard error.


2. Training-optimal allocation

Minimizing (1) subject to $$6ND = C_{\text{train}}$$ with multiplier $$\lambda$$:

$$ \mathcal{L}(N, D, \lambda) = E + A N^{-\alpha} + B D^{-\beta} + \lambda \left( 6 N D - C_{\text{train}} \right) \tag{5} $$

The first-order conditions are

$$ \alpha A N^{-\alpha} = 6 \lambda N D, \qquad \beta B D^{-\beta} = 6 \lambda N D \tag{6} $$

and eliminating $$\lambda$$ gives the condition along the training-optimal path:

$$ \alpha A \left(N^\right)^{-\alpha} = \beta B \left(D^\right)^{-\beta} \tag{7} $$

Substituting $$D^* = C_{\text{train}} / (6 N^*)$$ yields the power laws

$$ N^(C) = G \left( \frac{C}{6} \right)^{a}, \qquad D^(C) = G^{-1} \left( \frac{C}{6} \right)^{b} \tag{8} $$

$$ a = \frac{\beta}{\alpha + \beta}, \qquad b = \frac{\alpha}{\alpha + \beta}, \qquad G = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}} \tag{9} $$

At the fitted constants, $$a =$$ 0.453 and $$b =$$ 0.547: tokens should grow slightly faster than parameters. At $$10^{23}$$ FLOPs the training-optimal model has 14.8B parameters and 1.12T tokens (75.8 tokens per parameter) and reaches 2.005 nats, against 2.103 nats under Kaplan et al.'s parameter-heavy allocation.


3. Counting inference

Training-optimality ignores the cost of serving. Decoding costs $$2N$$ FLOPs a token, so a model that serves $$I$$ tokens over its life costs

$$ C_{\text{total}}(N, D; I) = 6 N D + 2 N I \tag{10} $$

To reach a target loss at least total cost, invert (1) for the tokens a model of size $$N$$ needs,

$$ D(N; L_{\text{target}}) = \left( \frac{B}{L_{\text{target}} - E - A N^{-\alpha}} \right)^{1/\beta} \tag{11} $$

and minimize the one-variable objective over $$N > \left( A / (L_{\text{target}} - E) \right)^{1/\alpha}$$:

$$ C_{\text{total}}(N) = 6 N , D(N; L_{\text{target}}) + 2 N I \tag{12} $$

Setting $$\mathrm{d}C_{\text{total}} / \mathrm{d}N = 0$$ gives

$$ \alpha A \left(N^\right)^{-\alpha} - \beta B \left(D^\right)^{-\beta} = \frac{\beta B \left(D^*\right)^{-\beta - 1}}{3} , I \tag{13} $$

For $$I > 0$$ the right side is positive, so the balance (7) tips: the capacity penalty is left larger than the data penalty. The lifecycle-optimal model is smaller than the training-optimal one and over-trained, at a much higher token-to-parameter ratio.


4. Results across budgets

Table 1 sets the two allocations side by side for $$I = 100\text{T}$$ tokens served, each budget's training-optimal loss as the target.

Table 1: Training-optimal and lifecycle-optimal allocations at 100T tokens served.

Budget $$C_{\text{train}}$$Target lossTraining-optimal $$N^$$ / $$D^$$Lifecycle-optimal $$N^$$ / $$D^$$Tokens per parameterLifecycle compute saved
$$10^{20}$$2.600647M / 25.8B94M / 4.58T48,81083.5%
$$10^{21}$$2.3291.8B / 90.7B303M / 6.49T21,42680.3%
$$10^{22}$$2.1395.2B / 319B1.0B / 9.47T9,29975.2%
$$10^{23}$$2.00514.8B / 1.12T3.6B / 14.4T4,00566.4%
$$10^{24}$$1.91242.1B / 3.96T13.4B / 23.3T1,73851.6%

The saving falls as the budget grows, because a fixed 100T tokens of serving weighs less against a larger training run.

Conclusions

  1. Data grows a little faster than size. Across the 6 budgets, $$a =$$ 0.453 and $$b =$$ 0.547, close to the equal split Hoffmann et al. report.
  2. Serving changes the answer. At $$10^{23}$$ FLOPs and 100T tokens served, a 3.6B model at 4,005 tokens per parameter cuts lifecycle compute by 66.4%.
  3. Plan a model by its life. Every figure here can be recomputed for another budget or serving volume on the Stage, and traced to its script in the Model tab.