Multi-Teacher Online Distillation: Global vs. Domain-Level Token Averaging
When you train a single student model across multiple capability areas at once—math, coding, dialogue, instruction following—multi-teacher online distillation lets task-specific teachers give feedback on the student's own generations. But naively combining teacher losses introduces a hidden length bias: domains with longer responses end up with larger gradient weights, and the student skews toward them. This article explains why global token averaging (GT) amplifies response-length bias, how domain-level token averaging (DT) rebalances influence at the domain level, how each choice affects gradients and final capabilities, and practical steps for measuring, configuring, and monitoring domain weights. You will learn how to pick and tune an aggregation strategy so the student becomes a genuinely balanced multi-capability model rather than one driven by response length and default settings.
When training across several capability areas at the same time, a common approach is to first train or align relatively specialized teacher models for tasks such as math, coding, dialogue, and instruction following, and then distill those capabilities into a single unified student model. This class of methods typically falls under online policy distillation: the student first generates a response, the teacher for the corresponding domain gives feedback on that response, and the student updates its parameters accordingly. Because the feedback targets the student’s own current output rather than a fixed offline dataset, the learning signal stays close to the student’s present capability distribution.
But multi-teacher online distillation is not simply a matter of summing the losses from several teachers. The problem that is easy to overlook is that different domains differ in response length, number of samples, and token composition. If the aggregation method is poorly designed, some domains may end up with a larger gradient weight merely because their responses are longer, ultimately causing the student model to become lopsided.
1. What Is Multi-Teacher Online Distillation, and Why Aggregation Matters
The core of multi-teacher online policy distillation can be summarized in three steps:
- The student model generates a response under its current policy.
- Based on the task domain, the corresponding teacher model evaluates or gives feedback on that response.
- The student model uses the feedback to compute a loss and update its parameters.
“Online” here means the teacher’s feedback is on what the student currently generates, not on fixed samples from an offline dataset. The benefit is that the signal stays close to the student’s real distribution and supports continuous correction. But it also raises an engineering problem: when several domains participate in training at once, the losses must be aggregated into a single scalar for backpropagation.
If the aggregation is unreasonable, you get a situation that looks fair but isn’t: every domain participates in the loss calculation, yet each domain’s influence on the final parameter update is not the same.
2. The Core Pitfall: Why “Averaging” Does Not Equal “Fairness”
The most intuitive approach is global token averaging. Sum the losses over all domains and all valid tokens, then divide by the total token count. This is easy to implement, but it automatically introduces a length bias.
Suppose math responses are on average long, while instruction-following responses are on average short. Under global token averaging, the math task occupies a larger share of the total loss because it contributes more tokens. Even if the average loss per token is similar, the domain with more tokens receives a larger overall weight.
An intuitive example: one student’s response contains hundreds of tokens, while another’s contains only a few dozen. If you mix all token losses together and average them, the former contributes more loss terms and therefore has a larger influence on the total loss.
This is the hidden danger of global token averaging: response length indirectly becomes weight. A longer domain is not necessarily more important, yet under the default aggregation it naturally gets more of a say.
So “averaging” does not automatically mean fairness. The real question is whether we want all tokens treated equally, or each domain treated equally first and then averaged within the domain.
3. Two Aggregation Strategies: Global Token Averaging vs. Domain-Level Token Averaging
To make the difference clearer, let us introduce some basic notation.
Suppose a training batch contains several domains, with the domain set denoted D and the number of domains participating in this batch being |D|. Let n_d be the number of valid tokens for domain d in this batch, and let N be the total number of valid tokens across all domains, so N = sum(n_d).
Let l_i be the distillation loss at each valid token position. Here l_i can represent the discrepancy between the student and the teacher’s feedback at that position—for example, a KL divergence between output distributions, a policy distance, or another loss form suited to the training objective. The key point is that l_i is computed only on valid tokens; padding, truncation, and invalid positions should not be counted.
Global token averaging can be understood as first computing the total loss L_GT = (1 / N) * sum(l_i). Since the loss sum for domain d contains n_d tokens, the domain’s weight in the total loss is approximately:
w_d^GT = n_d / N
This means that the more valid tokens a domain contributes, the larger its share of the total loss. Domains with longer responses therefore naturally gain a higher weight.
Domain-level token averaging works differently. It first assigns weight at the domain level, then averages over tokens within each domain. By default, each domain can receive the same weight:
w_d^DT = 1 / |D|
Then the average loss within domain d is computed as:
(1 / n_d) * sum_{i in d}(l_i)
The total loss can then be written as:
L_DT = (1 / |D|) * sum_d [(1 / n_d) * sum_{i in d}(l_i)]
If the business wants a certain domain to matter more, the equal weighting can be replaced with a preset weight α_d, where sum(α_d) = 1. The total loss then becomes:
L_DT_weighted = sum_d [α_d * (1 / n_d) * sum_{i in d}(l_i)]
The key difference is this: global token averaging hands weight to the token count, while domain-level token averaging hands weight to the domain first and then averages within the domain. The former is more easily amplified by response length; the latter keeps better controllability at the domain level.
4. Example Code: How the Two Weighting Strategies Are Implemented
The pseudocode below shows the basic difference between the two strategies. In real engineering, you also need to handle empty domains, imbalanced batch sampling, masking, temperature, length truncation, and teacher selection.
def gt_loss(per_token_losses):"""Global token averaging: average directly over all valid tokens.per_token_losses is an already-masked list of per-token losses."""if not per_token_losses:return 0.0total = sum(per_token_losses)num_tokens = len(per_token_losses)# Longer responses contribute more tokens, so they are implicitly amplifiedreturn total / num_tokens
The implementation of global token averaging is very direct: all tokens are treated equally. But that is exactly where its problem lies. If a domain’s responses are clearly longer, it contributes more loss terms and therefore a larger weight.
Domain-level token averaging first needs to aggregate by domain, then combine domains with weights across domains.
def dt_loss(per_token_losses, domain_ids, fixed_weights=None):"""Domain-level token averaging: first aggregate by domain, then averagewithin each domain, and finally combine across domains with weights.per_token_losses: flattened valid-token lossesdomain_ids: domain labels corresponding one-to-one with per_token_lossesfixed_weights: optional dict, e.g., {0: 0.4, 1: 0.3, 2: 0.2, 3: 0.1}"""domain_sums = {}domain_counts = {}for loss, d in zip(per_token_losses, domain_ids):domain_sums[d] = domain_sums.get(d, 0.0) + lossdomain_counts[d] = domain_counts.get(d, 0) + 1# Keep only domains that actually appear in this batch and have at least 1 valid tokenactive_domains = [d for d in domain_sums if domain_counts[d] > 0]if not active_domains:return 0.0if fixed_weights is None:# Default to equal domain weights, but normalize by the number of domains actually presentweights = {d: 1.0 / len(active_domains) for d in active_domains}else:# If preset domain weights are given, re-normalize only over domains actually presentdenom = sum(fixed_weights.get(d, 0.0) for d in active_domains)if denom <= 0:# If the preset weights cannot cover the batch's domains, fall back to equal weightsweights = {d: 1.0 / len(active_domains) for d in active_domains}else:weights = {d: fixed_weights.get(d, 0.0) / denomfor d in active_domains}total = 0.0for d in active_domains:domain_mean = domain_sums[d] / domain_counts[d]total += weights[d] * domain_meanreturn total
There is an edge case that is easy to overlook: if a domain is not sampled in this batch, you must not still use the “preset total number of domains” as the denominator. Otherwise the total loss is scaled down overall and becomes incomparable across batches. A safer approach is to re-normalize only over the domains that actually appear in this batch, or to explicitly log that the domain is missing from this batch.
Also note that domain-level token averaging blocks cross-domain length amplification, not within-domain length effects. Within a domain, if you still average per token, long responses still contribute more token terms to the average. So the more accurate statement is this: domain-level token averaging reduces the cross-domain squeezing of long-answer domains over short-answer domains, but it does not automatically solve the within-domain long-text bias.
5. GT or DT: How to Choose a Weighting Strategy for Multi-Teacher Distillation
Neither method is absolutely better; the choice depends on the training objective and business priorities.
| Aspect | Global Token Averaging (GT) | Domain-Level Token Averaging (DT) |
|---|---|---|
| Weight source | Allocated by number of valid tokens | Domain weight first, then within-domain averaging |
| Affected by response length | Clearly affected | Not directly amplified by response length at the domain level |
| Needs preset domain weights | No | Usually yes; equal domain weights by default |
| Suitable scenarios | When all tokens should be equal and long-answer tasks are naturally important | When multiple capabilities should stay balanced and short-answer tasks must not be drowned out |
| Main risk | Long-answer domains crowd out short-answer domains | Short tasks may be relatively amplified; small-sample domains fluctuate more |
| Missing-domain handling | Changes naturally with the tokens that appear | Must re-normalize over domains actually present to avoid shrinking the total loss |
If the training goal is to improve long-chain reasoning, complex code generation, or long-form math solutions, and the business genuinely wants long-answer tasks to take a larger share, then global token averaging can serve as a baseline. Even so, it is advisable to measure each domain’s weight first and confirm that the length bias matches expectations.
If the training goal is for the model to be simultaneously capable at math, coding, dialogue, and instruction following, with no clear preference for any long-answer domain, then domain-level token averaging is usually the safer choice. It prevents short-answer domains—such as format following, safety alignment, and simple instruction execution—from being naturally drowned out by long-answer domains.
A more sensible approach is to make domain weights configurable. Do not hardcode them as “all domains equal” or “all tokens equal”; instead, set α_d based on business goals. For example, if you value instruction following more, raise that domain’s weight; if you value math reasoning more, raise the math domain’s weight. The key is to make the weights explicit hyperparameters rather than something implicitly determined by response length.
6. How the Weighting Method Affects Gradients and Final Capabilities
From the perspective of parameter updates, the loss aggregation method directly changes how the gradient scale is allocated.
Under global token averaging, domain d’s contribution to the total gradient can be approximated as:
(1 / N) * sum_{i in d} ∇l_i
If the domain has more valid tokens n_d, its gradient sum contains more terms. Even if the average gradient magnitude per token is similar, a long-answer domain can obtain a larger overall gradient scale and thus occupy a more prominent position in a single update step.
Under domain-level token averaging, domain d’s contribution is closer to:
α_d * (1 / n_d) * sum_{i in d} ∇l_i
Here the within-domain average first compresses the effect of token count on the gradient scale, and then α_d controls the relative priority between domains. In this way, domain importance is determined by an explicit weight rather than indirectly by response length.
Keep in mind that the gradient norm is not determined by the aggregation method alone; it is also affected by loss scale, temperature, teacher distribution, the optimizer, gradient clipping, learning rate, and other factors. So you cannot simply assume that using domain-level token averaging guarantees every domain improves evenly. At the very least, it reduces the hidden bias of “length equals say.”
From a capability-configuration perspective, the aggregation method is itself a capability configurator. The default global token averaging may make the model more of a “long-answer expert,” while domain-level token averaging is closer to “balanced multi-domain training.” If you want the student model to become a balanced, general-purpose model, you need to carefully design the domain weights.
7. Practical Advice: From Measurement to Configuration and Monitoring
When putting this into practice, you can follow the process below.
Measure first, then choose the aggregation strategy.
Before formal training, sample a batch of responses from each domain and compute the average valid token length after masking. If a domain’s average length is clearly longer, estimate its weight share under global token averaging:w_d = n_d / sum(n_d). If a domain’s share is too high, it may implicitly dominate training.Without a clear priority, prefer domain-level token averaging.
If the business goal is simply “improve several capabilities” rather than “one long-answer task is especially important,” start with domain equal weights or a lightly configured domain-level token averaging. This effectively prevents a long-answer domain from dominating.Make domain weights configurable hyperparameters.
Do not hardcode the weights. Put α_d in a config file or an experiment-management layer so that different configurations can be compared. For example, set the weights for math, coding, dialogue, and instruction following to 0.3, 0.3, 0.2, and 0.2 respectively, then adjust based on validation-set performance.Handle missing domains within a batch.
If a batch contains no samples from a certain domain, do not directly reuse the preset all-domain denominator. Instead, re-normalize only over the domains that actually appear in this batch, or log the missing domain. Otherwise the total loss scale changes from batch to batch and affects training stability.Monitor per domain, not just the total loss.
A dropping total loss does not mean every capability is improving. Monitor math accuracy, code pass rate, instruction-following rate, format compliance, safety alignment metrics, and so on separately. If a domain’s metric stalls or degrades, check that domain’s weight, sampling ratio, teacher feedback quality, and data difficulty.Also pay attention to temperature, teacher feedback distribution, and loss type.
Distillation signal quality depends not only on the weighting method but also on the shape of the teacher’s output distribution. Temperature settings, teacher model selection, whether to use top-k truncation, and whether to use KL or another policy loss all affect the gradient direction. The weighting strategy should be tuned together with these design choices.Define the risk boundaries.
The conclusions above depend on the specific implementation and data distribution. Domain-level token averaging is more balanced, but it may relatively amplify short-answer tasks, such as format constraints and safety alignment. For small-sample domains, batch-to-batch weight fluctuation may also be larger. Therefore, adjust using per-domain validation sets and training curves, and do not treat any single aggregation strategy as a permanent standard answer.
8. Conclusion
The key to multi-teacher online distillation is not just letting one student model learn from multiple teachers at once, but re-examining those settings that seem default yet actually affect the capability distribution. Global token averaging is simple to implement, but it lets response length implicitly become weight; domain-level token averaging emphasizes controllability at the domain level and suits balanced multi-capability training.
A truly reliable weighting strategy does not lump all weights together, but makes each domain’s influence measurable, configurable, and monitorable. Only when the aggregation method is aligned with the training objective is the student model more likely to become a genuine multi-capability model, rather than one led astray by response length and default settings.
