Parametric Scaling Laws and Inference-Aware Compute Allocation for Autoregressive Transformers

Abstract. Empirical scaling laws govern how autoregressive language model cross-entropy loss improves with model parameter count $$N$$ and training token volume $$D$$. While classical compute-optimal scaling ("Chinchilla optimality") minimizes validation loss under a fixed pre-training FLOP budget $$C_{\text{train}} \approx 6ND$$, production deployments must minimize total lifecycle compute across both training and inference.

We fit a parametric scaling surface across 90 empirical IsoFLOP training runs spanning 6 orders of magnitude in compute ($$10^{18}$$ to $$10^{22}$$ FLOPs), derive closed-form exponents for training-optimal allocation, and solve the constrained lifecycle Lagrangian when a model serves $$I$$ lifetime inference tokens. At $$C_{\text{train}} = 10^{23}$$ FLOPs and $$I = 100\text{T}$$ inference tokens, shifting from a Chinchilla-optimal 14.8B model to an inference-optimal 3.6B model trained on 14.4T tokens achieves the identical validation loss (2.005 nats) while reducing total lifecycle FLOPs by 66.4%.

Open the interactive compute-frontier simulator on the Stage

1. Parametric Loss Decomposition

Consider a dense autoregressive transformer with $$N$$ non-embedding parameters trained for a single epoch on $$D$$ tokens sampled i.i.d. from a natural language corpus. Following Hoffmann et al.p. 7, we model the validation cross-entropy loss $$L(N, D)$$ (in nats per token) as the sum of an irreducible entropy floor $$E$$ and two additive power-law finite-approximation penalties:

$$ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1} $$

where $$E > 0$$ denotes the intrinsic entropy of natural text as $$N, D \to \infty$$, the term $$A N^{-\alpha}$$ captures the functional approximation error of a finite network of capacity $$N$$, and $$B D^{-\beta}$$ captures the statistical estimation error after observing $$D$$ training tokens.

Under standard dense transformer FLOP accounting (one forward pass of $$2N$$ FLOPs and one backward pass of $$4N$$ FLOPs per token), pre-training compute satisfies:

$$ C_{\text{train}}(N, D) = 6 N D \tag{2} $$

Non-Linear Least Squares Estimation

We estimate $$\theta = (E, A, B, \alpha, \beta)$$ via bounded non-linear least squares across our empirical sweep of $$M = $$ 90 training runs:

$$ \hat{\theta} = \underset{\theta \in \Theta}{\arg\min} \sum_{i=1}^{M} \left( L_i - E - \frac{A}{N_i^{\alpha}} - \frac{B}{D_i^{\beta}} \right)^{2} \tag{3} $$

Fitting Equation (3) across all six IsoFLOP budgets recovers the empirical loss surface:

$$ \hat{L}(N, D) = 1.692 + \frac{401.9}{N^{0.339}} + \frac{422.5}{D^{0.281}} \tag{4} $$

The fitted constants from scripts/fit_scaling_laws.py are $$E = $$ 1.692, $$A = $$ 401.9, $$B = $$ 422.5, $$\alpha = $$ 0.339, and $$\beta = $$ 0.281.

2. Lagrangian Derivation of Training-Optimal Scaling

To minimize validation loss $$L(N, D)$$ subject to a fixed training compute budget $$6ND = C_{\text{train}}$$, we form the Lagrangian with multiplier $$\lambda$$:

$$ \mathcal{L}(N, D, \lambda) = E + A N^{-\alpha} + B D^{-\beta} + \lambda \left( 6 N D - C_{\text{train}} \right) \tag{5} $$

Setting the first-order partial derivatives with respect to $$N$$ and $$D$$ to zero yields the stationary conditions:

$$ \frac{\partial \mathcal{L}}{\partial N} = -\alpha A N^{-\alpha - 1} + 6 \lambda D = 0 \quad \Longrightarrow \quad \alpha A N^{-\alpha} = 6 \lambda N D \tag{6a} $$

$$ \frac{\partial \mathcal{L}}{\partial D} = -\beta B D^{-\beta - 1} + 6 \lambda N = 0 \quad \Longrightarrow \quad \beta B D^{-\beta} = 6 \lambda N D \tag{6b} $$

Equating Equations (6a) and (6b) eliminates $$\lambda$$ and establishes the optimality condition along the training-optimal expansion path:

$$ \alpha A \left(N^\right)^{-\alpha} = \beta B \left(D^\right)^{-\beta} \tag{7} $$

Substituting $$D^* = C_{\text{train}} / (6 N^*)$$ into Equation (7) yields the closed-form power laws for training-optimal model size and training tokens:

$$ N^(C_{\text{train}}) = G \left( \frac{C_{\text{train}}}{6} \right)^{a}, \qquad D^(C_{\text{train}}) = G^{-1} \left( \frac{C_{\text{train}}}{6} \right)^{b} \tag{8} $$

where the scaling exponents $$a, b$$ and proportionality constant $$G$$ are given by:

$$ a = \frac{\beta}{\alpha + \beta}, \qquad b = \frac{\alpha}{\alpha + \beta}, \qquad G = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}} \tag{9} $$

Evaluating Equation (9) at our fitted parameters gives $$a = $$ 0.453 and $$b = $$ 0.547. At $$C_{\text{train}} = 10^{23}$$ FLOPs, the training-optimal model allocates 14.8B parameters and 1.12T training tokens (75.8 tokens per parameter), achieving a validation loss of 2.005 nats compared to 2.103 nats under Kaplan et al.'s parameter-heavy allocation.

3. Inference-Aware Lifecycle Optimization

Pre-training optimality ignores the computational cost of serving the trained model. Let $$I$$ denote the total number of tokens generated by the model over its deployment lifetime. Since autoregressive decoding requires $$2N$$ FLOPs per token, total lifecycle compute is:

$$ C_{\text{total}}(N, D; I) = 6 N D + 2 N I \tag{10} $$

To achieve a target validation loss $$L(N, D) = L_{\text{target}}$$ at minimum total lifecycle compute, we invert Equation (1) to express required training tokens $$D$$ as a function of $$N$$:

$$ D(N; L_{\text{target}}) = \left( \frac{B}{L_{\text{target}} - E - A N^{-\alpha}} \right)^{\frac{1}{\beta}} \tag{11} $$

Substituting Equation (11) into Equation (10) yields the unconstrained single-variable lifecycle objective over the feasible domain $$N > \left( A / (L_{\text{target}} - E) \right)^{1/\alpha}$$:

$$ C_{\text{total}}(N; L_{\text{target}}, I) = 6 N \left( \frac{B}{L_{\text{target}} - E - A N^{-\alpha}} \right)^{\frac{1}{\beta}} + 2 N I \tag{12} $$

Differentiating Equation (12) with respect to $$N$$ and setting $$\mathrm{d}C_{\text{total}} / \mathrm{d}N = 0$$ gives the inference-adjusted optimality condition:

$$ \beta B \left(D^\right)^{-\beta} - \alpha A \left(N^\right)^{-\alpha} = \frac{\beta B \left(D^*\right)^{-\beta - 1}}{3} , I \tag{13} $$

Whenever $$I > 0$$, the right-hand side of Equation (13) is strictly positive, implying $$\beta B (D^)^{-\beta} > \alpha A (N^)^{-\alpha}$$: the lifecycle-optimal architecture is systematically smaller ($$N^ < N_{\text{Chinchilla}}$$) and over-trained on a much higher token-to-parameter ratio*.

4. Empirical Comparison Across Scaling Regimes

Table 1 summarizes the optimal model architecture, token budget, and lifecycle compute efficiency across five target quality tiers when serving $$I = 100\text{T}$$ lifetime inference tokens.

Table 1: Comparison of Chinchilla Training-Optimal (2022) and Inference-Aware Lifecycle-Optimal (2024) allocations across compute scales.

Key Takeaways for Frontier Model Design

1. Equal-exponent training scaling holds across six orders of magnitude: Non-linear curve fitting across all 90 IsoFLOP runs yields $$a = $$ 0.453 and $$b = $$ 0.547, confirming that training token budgets must grow slightly faster than parameter counts.
2. Inference dominates lifecycle compute for widely deployed models: When serving $$I = 100\text{T}$$ tokens at the $$10^{23}$$-FLOP quality tier ($$L^* = $$ 2.005 nats), over-training a 3.6B model to 4,005 tokens per parameter cuts total training + inference compute by 66.4%.
3. Interactive verification: Every equation, parameter, and IsoFLOP curve in this paper can be stress-tested live on the Stage or inspected directly in the Model tab.