Generative Recommenders: Assumptions and Reward Design Matter More Than Architecture
Recently published generative recommendation systems offer plenty of material for discussion in architecture design, reinforcement learning pipelines, and reward modeling. Simply restating highlights like "generative ranking," "end-to-end modeling," and "reinforcement learning optimization" risks missing the more fundamental questions: What determines the performance ceiling of these systems? Does the reward signal truly represent user value? Is reward definition still the core challenge in recommendation? Rather than recapping the technical details, this article examines generative recommenders through three lenses—assumptions, rewards, and evaluation—focusing on the less flashy issues that ultimately decide a system's upper limit.
The Distillation Assumption Behind the Architecture
Generative recommendation systems typically model recommendation as a sequence generation or candidate reranking problem, using a generative model to directly learn the matching relationship between user interests and candidate content. The architecture genuinely has its strengths: stronger contextual modeling, more unified sequence modeling, and the potential to optimize long-term returns via reinforcement learning. But an important question follows: where does the generative model’s supervision signal come from?
A common implementation uses the fused ranking score from an upstream fine-ranking model or interest modeling model as a teacher signal, having the generative model perform distillation-style fitting. The benefits are straightforward: the teacher signal typically already integrates multiple metrics such as click-through rate, long-view rate, like rate, and negative feedback. As long as the student model fits well, it achieves fairly stable offline gains. And since a larger student model has greater fitting capacity, a “bigger is better” scaling law naturally emerges.
However, this assumption imposes a clear boundary: if the reward or supervision signal depends entirely on the upstream ranker’s output, the generative model’s knowledge boundary is capped by the teacher. When the student model is small, scaling up significantly improves distillation quality; but once student capacity approaches the teacher model or the information ceiling of the training data, the marginal returns of further scaling diminish. In other words, a generative model cannot automatically surpass the “value boundary” defined by the teacher signal. Breaking through this limitation requires introducing value signals independent of the teacher—such as long-term retention, session depth, negative experiences, exploration feedback, or causally corrected user value—rather than simply doing stronger function fitting.
An illustrative training objective can be abstracted as:
for batch in data_loader:teacher_score = upstream_ranker.score(user, context, candidates)student_score = generative_recommender(user, context)loss = distill_loss(student_score, teacher_score) \+ lambda1 * direct_value_loss(student_score, long_term_feedback) \+ lambda2 * regularization(student_score)update(student_score, loss)
Whether direct_value_loss can be constructed—and how—determines whether the student model can break past the teacher’s boundary. Without this term, the system’s ceiling will most likely be constrained by the teacher model.
Does a Fused Reward Score Equal User Value?
Such systems typically define a personalized fused reward score that combines click-through rate, long-view rate, like rate, and similar metrics as the optimization objective. The question is: does combining the AUCs or ranking scores of multiple metrics equate to genuine user value? Not necessarily.
First, metric combinations usually rely on static weights or dynamic weights under limited conditions, making it hard to capture value differences across users, scenarios, and content supply. Second, offline metrics are affected by exposure selection bias: the model only sees samples exposed by historical policies, so directly optimizing a combined AUC can amplify existing biases. Third, short-term behaviors like clicks and long views are not monotonically consistent with long-term retention and user satisfaction—over-optimizing short-term metrics can encourage clickbait, low-quality content, or negative experiences. Finally, recommendation systems involve feedback loops: whatever the model optimizes is what users see more of, which in turn changes future data distributions. Relying solely on offline AUC combinations makes it very hard to assess long-term impact.
Therefore, a fused reward score should be treated as a “proxy for business metrics,” not as user value itself. A more sound approach incorporates debiasing corrections, causal evaluation, long-term experiments, and multi-objective constraints. For example, the reward can be decomposed into immediate behavioral gains, session value, long-term retention value, and negative-experience penalties, then validated online through long-running A/B tests. In pseudocode:
reward = w1 * f(click) \+ w2 * f(long_view) \+ w3 * f(session_depth) \+ w4 * f(retention_proxy) \- w5 * f(negative_feedback)
The key is not the formula itself, but whether the weights are learnable, user-segmented, bias-corrected, and backed by long-term metrics. If the reward is just an AUC combination of a few short-term metrics, both the system’s ceiling and the user experience gains will likely be limited.
Reward Definition Remains the Core Problem
The fundamental question in recommendation has always been: how do you define the reward, or user value? Upstream pretraining or foundation tasks typically focus on basic behavior estimation—click-through rate, completion rate, like rate, long-view rate, and so on—while downstream ranking or policy optimization focuses on combining these base metrics into overall watch time, long-term retention, or long-term value. Whether that combination is done scientifically directly determines a recommender’s ceiling.
The prevailing approach today is multi-objective fusion: train multiple behavior prediction models offline, then combine them through weighted averaging, ranking, or constrained optimization. This works from an engineering standpoint but faces three challenges: the gap between proxy metrics and true value; fusion weights that depend heavily on manual tuning; and long-term value feedback that is sparse and delayed, making it hard to optimize directly. A more systematic approach is hierarchical reward modeling: a base behavior layer estimates immediate feedback accurately; a session value layer models continuous consumption and exploration; a long-term value layer leverages sparse signals like retention, return visits, and satisfaction; and a negative-experience layer applies separate constraints to prevent short-term metrics from harming user experience. Reinforcement learning can optimize long-term returns within this framework—but only if the reward model is reliable enough; otherwise, it simply amplifies biases in the reward definition.
Lessons from the LLM Playbook
Looking back at the success of LLMs, one of the key breakthroughs was instruction tuning combined with reinforcement learning from human feedback (RLHF). Two conditions underpinned that success: the reward model’s scale was roughly comparable to the pretraining model’s, and the reward definition was relatively clear—human preference, helpfulness, safety, and so on. A clear reward plus sufficient reward-modeling capacity allowed the model to optimize consistently across a complex space.
Recommendation systems differ markedly from this path. Their rewards tend to be small in scale, built from existing business metric combinations, with noisy and delayed feedback, and user value is highly personalized and context-dependent. Many systems still use short-term behavioral metrics as the primary reward, lacking a genuinely clear, scalable definition of user value. As a result, reinforcement learning in recommendation often falls into the trap of “optimizing proxy metrics”: offline metrics improve, but long-term experience may not.
So if generative recommendation wants an LLM-style breakthrough, the focus cannot be solely on generative architectures or RL algorithms. What is needed is a large-scale, well-defined, and evaluable reward modeling system. Within compliance and privacy constraints, explicit feedback, long-term behavior, negative feedback, and exploration data can be used to build more reliable reward models, closed-looped with offline policy evaluation, simulation-based evaluation, and long-term online experiments.
Practical Recommendations for Implementation
First, decouple the teacher signal from the student’s objective. Distillation can serve cold starts and stabilize training, but a direct value-optimization channel should be preserved, with monitoring to detect whether the student is merely fitting the teacher. Second, make reward definitions evaluable. Any fused reward score should be paired with debiased evaluation, long-term metrics, and negative-experience monitoring, so that short-term AUC gains don’t mask long-term losses. Third, use reinforcement learning with caution. When the reward is unreliable, RL amplifies bias; ensure the reward model and evaluation system are solid before introducing policy optimization. Fourth, take exploration and feedback loops seriously. Generative recommenders are more expressive, but that also makes filter bubbles easier to form—diversity, exploration, and negative-feedback constraints should be built into the reward.
Overall, the technical highlights of generative recommendation are worth attention, but what truly determines its ceiling is not the generative architecture itself—it is the supervision signal and the reward definition. Fine-ranking distillation delivers stable gains but sets a knowledge boundary; fused reward scores are implementable but are not user value. The next stage of competition in recommendation systems may not be about “whether the model is generative,” but about “who can define user value more clearly, more scalably, and more verifiably.”
