Applying LLMs and VLMs in Recommendation Systems: Long-Sequence Modeling, Generative Reranking, and Multimodal Understanding
Recommendation systems must surface the content a user is most likely to engage with from a massive candidate pool and order it sensibly. This article examines how large language models (LLMs) and vision-language models (VLMs) can be applied across four areas: end-to-end long-sequence modeling, sequence compression, generative reranking, and multimodal content understanding. It explains why these models fit recommendation, how to make long sequences computationally feasible, how constrained decoding keeps generative reranking valid, and how multimodal models help with cold-start items. It closes with engineering guidance on balancing effectiveness, latency, and cost—emphasizing tail-latency measurement with percentiles, offline/online consistency, and a phased rollout.
Why Recommendation Systems Are Going “Large-Model”
The goal of a recommendation system is to quickly find the items a user is most likely to be interested in from a massive pool, and to order them for display. Mainstream implementations mostly use a multi-stage cascade:
| Stage | Candidate scale (order of magnitude) | Core constraint | Primary goal |
|---|---|---|---|
| Retrieval | Millions–billions → thousands | Millisecond latency, very high throughput | High coverage, low miss rate |
| Pre-ranking | Thousands → hundreds | Lightweight model | Balance accuracy and compute |
| Ranking | Hundreds → tens | Complex feature crossing | Fine-grained modeling of user interest |
| Reranking | Tens | Overall list payoff | Context awareness, diversity, within-list interaction |
Reranking sits at the end of the funnel and directly determines what is finally shown, significantly affecting core metrics such as click-through rate and dwell time. Display formats such as carousels and infinite-scroll feeds are especially sensitive to ordering.
Over the past decade and more, recommendation models have evolved from logistic regression and factorization machines to deep neural networks and attention mechanisms. Fundamentally, they have been doing the same thing: predicting the user’s next interaction (click, watch, favorite, etc.) based on their history and context. This is mathematically isomorphic to a language model “predicting the next word from the preceding context.” Large models bring not a single trick but three paradigm-level shifts:
- Scaling: data, parameters, and sequence length grow together, and performance keeps improving with investment;
- Unified representation: large pretrained models turn unstructured content into general-purpose semantic vectors;
- Unified architecture: prediction, retrieval, and even reranking can all be written as “sequence modeling + autoregressive generation.”
This article covers four applications of LLMs and multimodal large models (VLMs) in recommendation: long-sequence modeling, sequence compression, generative reranking, and multimodal content understanding. It then discusses, from an engineering perspective, how to balance effectiveness, latency, cost, and consistency.
Direction 1: End-to-End Long-Sequence Modeling
2.1 The bottleneck: the expressive ceiling of traditional sequence features
Early ranking models compress a user’s behavior sequence into a fixed-length vector—such as a mean or attention-pooled result—or keep only the most recent few dozen behaviors. This has two problems:
- Lossy information: compressing thousands or even tens of thousands of historical behaviors into a few dozen dimensions makes subtle interest drift hard to preserve;
- Decoupled from candidates: once the compressed result is written to feature storage, all candidates share the same representation. A single user vector must serve hundreds or thousands of candidate scores at once, and cannot dynamically select relevant history for a specific candidate.
Target attention improves the second point: at scoring time, the candidate item’s vector “queries” the history sequence, so only relevant history receives high weight. But as long as sequence length has a cap, the amount of queryable information is limited. So we need an approach that feeds the raw long sequence directly into the model, letting candidates truly “see” the complete history. Industry practice has already extended single-user sequences to tens of thousands in length.
2.2 Two-stage retrieval vs. end-to-end: two approaches
One common approach is “two-stage retrieval.” First, a lightweight model filters the most relevant behaviors from the very long sequence; then these are passed to the main model for scoring—similar to the “retrieve first, then generate” split in retrieval-augmented generation. Its advantage is simple implementation; its drawback is that filtering errors cannot be fully eliminated in end-to-end training.
The end-to-end approach lets the model directly consume the complete sequence, relying on two key enablers for feasibility:
- Sparsification: attention cost scales quadratically with sequence length, so longer sequences cost more. Structures such as local windows and sparse attention can significantly reduce computation;
- Inference optimization: on-demand loading of sparse embeddings, chunked processing, and incremental feature caching let a single forward pass cover a long context.
The cost of the end-to-end approach is a clear rise in training and inference complexity, which must be offset with engineering techniques—see the compression ideas in Section 4.
2.3 A general code skeleton for long-sequence modeling
# Illustrative pseudocode: expresses the modeling flow only, not any specific system interfacedef build_input(user, candidates, max_len=10_000):# 1. Take the user's raw behavior sequence (clicks, completions, favorites, etc.), not a pre-aggregated vectorraw_seq = user.raw_behavior_stream(max_len)# 2. Semantic vectors from the multimodal content-understanding module can be concatenated with behaviors by timestampitem_vecs = [encode(item) for item in raw_seq]# 3. Lightweight relevance filtering of the target candidate against sequence elements (optional, to control compute)pruned = [v for v in item_vecs if relevance(candidates[0], v) > 0]if len(pruned) < max_len:pruned += [PAD] * (max_len - len(pruned))# 4. Sequence modeling yields a user-context representation, then interaction scoring against each candidatectx = sequence_encoder(pruned) # long-sequence encodingscores = [rank_head(ctx, encode(c)) for c in candidates]return scores
The key to this code is steps 1 through 3: replacing pre-aggregated features with the raw behavior stream, and completing filtering and padding before feeding the encoder. Training and online serving must share the same preprocessing logic, or model performance will degrade sharply after launch.
Direction 2: Making Long Sequences “Computable” Through Compression
The real obstacle to deploying long-sequence solutions is compute and latency. The more candidates and the longer the sequence, the worse the redundant computation: the same user’s same history gets re-encoded hundreds or thousands of times. Industry commonly uses three types of countermeasures.
3.1 Caching intermediate results
The candidate-independent part of sequence encoding only needs to be computed once and cached for reuse. When the user’s behavior stream changes, the cache is synced via incremental updates.
The cost is storage overhead and version management. If the behavior stream updates in real time and the long sequence must be recomputed for every scoring call, that is impractical. So the first step to reduce cost is to split sequence encoding into a “candidate-independent” part and a “candidate-dependent” part, computing only the latter per candidate.
3.2 “Instance-as-dimension” style in-batch compression
In traditional implementations, the number of candidates shows up as the batch dimension: a batch of N candidates means N forward passes. Another approach folds candidates into an extra tensor dimension, so a single forward pass scores multiple candidates at once while sharing the encoding of the history sequence.
An analogy: this is like how a vision-language model processes multiple images—one main image with several reference images, rather than sending each image through the model separately. The benefits are threefold:
- the history sequence is encoded once and reused across multiple candidates;
- a single computation completes scoring for multiple candidates, with lower framework scheduling overhead;
- comparison information among in-batch candidates naturally enters the attention computation.
The cost is that intermediate tensors grow substantially, raising memory and communication overhead. Engineering requires finding a balance among sequence length, candidate count, and batch size.
3.3 Heterogeneous compute and mixed precision
- Dense computation (attention, fully connected layers) goes to high-throughput accelerators;
- Sparse queries (large embedding lookups) are handled by CPU memory or a distributed parameter server;
- Combined with mixed precision and quantization to lower the cost of a single forward pass.
These three ideas can be stacked. Whether to adopt them depends on whether the online latency budget can leave enough room for the long-sequence approach.
A Deeper Perspective: Compression Is Prediction
4.1 The equivalence from an information-theory view
The essence of lossless compression is continuously using the “known part” to predict the “unknown part.” Run-length encoding is the simplest example: consecutive repeated data is represented as “value + length,” and the longer the run, the higher the compression. Arithmetic coding goes further, assigning code lengths to each symbol with a probability model—the higher the probability, the fewer bits used. The key here is not the coding algorithm itself but the quality of the probability model.
Formally, if a probability model p gives the probability of a data segment x, the average bits needed to encode it is roughly:
L(x) ≈ −log₂ p(x) = −Σᵢ log₂ p(xᵢ | x₁, …, xᵢ₋₁)
This is exactly what a language model’s cross-entropy loss means (in bits). The more accurately the model predicts the next symbol, the lower the loss, and the more compressible the data.
This view has been validated in general domains: using a large language model as a probability estimator can directly compress text, images, and even audio, with compression rates significantly better than traditional lossless algorithms.
4.2 Implications for recommendation
A recommendation system predicts the user’s next action y (whether they click, etc.), with loss written as:
L = −Σ log p(yᵢ | user_contextᵢ, itemᵢ)
This is fully isomorphic to “predicting the next symbol from the preceding context.” The recommendation task is essentially modeling the user’s behavior sequence. From this we can draw three practical implications:
- More information, better compression: the more accurately we characterize the user’s context, the more accurately we predict the next action and the lower the model loss. This theoretically supports investing in long sequences and multimodal representations;
- Perplexity as a general proxy metric: perplexity can be defined as
exp(loss)—i.e., on average how many “equally likely choices” there are at each step. During training, perplexity can serve as an auxiliary monitoring metric decoupled from goals like click-through rate. If introducing a new feature keeps lowering perplexity without producing online gains, it often means the feature captures patterns unrelated to the business goal—such as user habits rather than current interest; - Compressibility is predictability: the difference in a model’s loss across different users and scenarios partly reflects how predictable those scenarios are. Traffic with persistently high loss may be exactly where cold-start or exploration strategies need reinforcement.
It’s worth emphasizing that the compression view is an understanding tool, not a new objective function. It cannot replace online experiments judged by business metrics.
Direction 3: Generative Reranking
5.1 From “scoring item by item” to “generating the whole list”
Traditional reranking scores each candidate independently, then adjusts the order via diversity constraints and interleaving rules. The problem is that a list is a whole: what goes in position 1 changes the optimal choice for position 2, and independent scoring cannot capture this interaction.
Generative reranking reformulates the problem as an autoregressive process, similar to a language model generating a sentence token by token:
P(list) = Πₜ P(item_t | user_context, item_1, …, item_{t−1})
At each step, the model selects the next most suitable item conditioned on “the already-generated sequence + user context,” equivalent to letting the model directly optimize the overall list payoff.
5.2 Constrained decoding: the key safeguard for generative approaches
The biggest risk of a generative flow is producing “illegal” results—duplicate items, or items outside the candidate pool. Common constraints include:
- Candidate-set constraint: decode only among real candidates;
- Deduplication constraint: mask already-generated items;
- Diversity constraint: limit consecutive appearances of the same category or author;
- Prefix constraint: fixed positions such as operations-driven forced insertions.
import numpy as np# Illustrative pseudocode: constrained decoding for generative rerankingdef generative_rerank(scorer, user_ctx, candidates, size):history = [] # generated sequence so far, used as context for subsequent decodingchosen = []for _ in range(size):scores = scorer.score(user_ctx, history, candidates)for j in history: # dedup: generated items no longer participate in decodingscores[j] = -np.inf# Diversity, category limits, forced slots, etc. can be extended herenxt = int(np.argmax(scores))chosen.append(candidates[nxt])history.append(nxt)return chosen
Compared with rule-based reranking, constrained decoding expresses business constraints like diversity and interleaving uniformly as “decoding-space restrictions.” The logic is more centralized, easier to test, and easier to iterate on.
5.3 Costs and control measures
Generative reranking is not a “free lunch”:
- Latency: step-by-step decoding means multiple forward passes. Reranking itself sits at the end of the pipeline with few candidates (usually tens), which is acceptable; but it is sensitive to P99 latency;
- Training objective: you need to build supervised samples of “given user context and preceding items.” How sequences are constructed (random prefixes, left-to-right sliding windows) directly affects what the model learns;
- Offline evaluation bias: the exposure data used for training is produced by an old policy, with position bias and popularity bias, so offline metrics may overestimate gains—online experiments must be the source of truth.
A safe way to roll out: run a small-scale pilot at the reranking stage first, use offline metrics to shortlist options and small-traffic online experiments to validate, then decide whether to expand to more scenarios.
Direction 4: VLMs Let Content Be “Understood”
Recommendation systems have long relied on item IDs and manual tags. New items have no behavior data, their ID vectors are nearly random, and cold start is hard. The value of multimodal large models is that they can understand the content itself with no behavior data at all.
6.1 A typical offline representation pipeline
Given the online latency budget, multimodal models are generally placed in offline or near-line pipelines, producing two kinds of assets:
- Semantic vectors: unified encoding of cover images, key frames, and titles, used jointly with item ID vectors;
- Structured tags: generated content themes, scenes, emotions, visual styles, etc., used for retrieval expansion and explainable recommendation.
# Illustrative pseudocode: offline content understandingdef enrich_item(item):frames = sample_frames(item.video, k=4) # sample key frames uniformlydesc = vlm.describe(frames + [item.title], # unified templateprompt="Output the content theme and visual style")item.semantic_tags = normalize(desc) # normalize to a controlled vocabularyitem.mm_vector = embed(frames, item.title) # multimodal vector for retrieval and rankingwrite_kv(item) # write to feature storage, query by ID online
6.2 Application patterns and cautions
- Cold start: new items first enter a candidate set close to existing items via semantic vectors, then gradually accumulate behavior data;
- Similar-item retrieval: semantic vectors can build an approximate-nearest-neighbor index as a complement to collaborative filtering;
- Search linkage: unified content-understanding artifacts can be reused in search and other scenarios, reducing duplicate build costs.
Three things to watch when rolling out:
- Version management of offline artifacts: after the multimodal model iterates, existing items need batch recomputation, so there should be clear version-switching and rollback mechanisms;
- Controlled vocabulary: freely generated tags tend to diverge and must be normalized to a controlled vocabulary, or they cannot be stably used as features;
- Cost-benefit matching: multimodal inference costs far more than pure text, so prioritize high-exposure or cold-start items rather than processing everything.
Engineering View: From “Can Do” to “Works Well”
7.1 Look at latency with percentiles, not means
Imagine a typical statistical trap. A team deploys a caching optimization, and online monitoring shows the mean latency rose from 112ms to 122ms (+9%), seemingly a negative optimization. Breaking the metric down further tells a completely different story:
| Metric | Before | After | Change |
|---|---|---|---|
| Mean | 112ms | 122ms | +9% |
| Median (P50) | 99ms | 54ms | −46% |
| P99 | ~200ms | ~400ms | ~+100% |
The principle behind the optimization: caching makes most requests hit and return fast; missed requests fall back to origin and take noticeably longer. The overall result is a distribution of “most faster, a few slower.”
The mean is extremely sensitive to a few extreme values. Suppose that of 100 requests, 99 drop from 100ms to 50ms and 1 rises from 100ms to 5050ms:
- Before: mean 100ms; after: mean 100ms—“looks unchanged”;
- In reality, P50 drops from 100ms to 50ms while P99 rises from 100ms to 5050ms.
That is, the mean can mask a hundreds-fold increase in tail latency. And the recommendation pipeline is a cascade, so tail latency at any terminal stage amplifies layer by layer, ultimately appearing as the lag users perceive.
Reliable monitoring approaches include:
- Track P50, P95, P99, and P99.9 together, and measure timeout rate separately;
- Compare the overall distribution shape before and after optimization with a cumulative distribution function (CDF), not just a single number;
- Use histograms to see whether the distribution becomes “bimodal,” i.e., hits and misses coexisting;
- Break down metrics by scenario, candidate count, sequence length, cache hit or miss, and so on.
import numpy as np# Illustration: multi-metric + histogram comparison of latency distributionsdef latency_report(name, ms):print(f"{name}: mean={np.mean(ms):.1f} p50={np.percentile(ms, 50):.1f} "f"p99={np.percentile(ms, 99):.1f} "f"timeout_rate={np.mean(np.array(ms) > 300):.2%}")counts, edges = np.histogram(ms, bins=[0, 50, 100, 200, 500, 1000, np.inf])for c, lo, hi in zip(counts, edges[:-1], edges[1:]):print(f" [{lo:.0f}, {hi if hi != np.inf else '∞'}) : {c}")
Conclusion: latency evaluation must return to percentiles and the full distribution. An optimization that only watches the mean may trade worse tail experience for a “better-looking” average.
7.2 Ensuring offline/online consistency
Large-model solutions have longer chains and more points where offline and online can diverge:
- Sequence construction: training uses the behavior stream visible at time T, while online uses the real-time behavior stream. You must avoid leaking future information into training samples (feature leakage);
- Content-understanding versions: the offline tagging version and the online usage version must be strictly aligned;
- Truncation rules: the filtering and truncation logic for long sequences must be exactly consistent on both sides.
A simple approach: have training samples and online requests share the same feature-production code, eliminating semantic drift between the two sides at the source.
7.3 Controlling cost
- Distillation: use a large model to produce soft labels offline, and train a lightweight model to handle online scoring;
- Quantization and mixed precision: lower per-inference cost while keeping accuracy loss controllable;
- Multi-level caching: cache user-level sequence encodings and item-level content vectors separately;
- Dynamic batching: use low-load periods to run offline recomputation and backfills.
7.4 A phased implementation roadmap
| Phase | Focus | Main output | Risk control |
|---|---|---|---|
| Phase 1 | Long-sequence features and content-understanding offline | Ten-thousand-scale sequence features, multimodal tags and vectors | Compare against baseline, confirm offline gains |
| Phase 2 | End-to-end long-sequence model + in-batch compression | Upgraded ranking model, latency targets met | Scale up only after percentile latency targets are met |
| Phase 3 | Generative reranking pilot | List-level optimization | Small-traffic experiments, guard against constraint failures |
| Phase 4 | Systematic operations | Monitoring, feedback loops, iteration loop | Interpretable metrics, rollbackable |
On deployment form, whether using self-built compute or a cloud-hosted solution, it is advisable to prioritize infrastructure with a complete capability chain: model training and inference services, containerized deployment, object storage, feature storage, and logging with monitoring and alerting. Cloud services such as Baidu AI Cloud typically cover these general capability categories; for specific specs and pricing, refer to the official documentation. A complete observability system lets effectiveness, latency, and cost problems be diagnosed within the same set of data.
Conclusion
LLMs and VLMs entering recommendation systems are not meant to replace the existing cascade architecture, but to inject three new capabilities:
- Longer memory: end-to-end long-sequence modeling lets the model read the complete history directly;
- More general understanding: multimodal large models give content a computable representation even at the cold-start stage;
- More holistic decisions: generative reranking treats the list as a whole for optimization.
All three come at the cost of compute and latency. Compression, distillation, quantization, caching, and similar techniques are essentially answering the same question: how to bring a large model’s generalization ability into a millisecond-scale online pipeline at controllable cost.
And from an information-theory view, recommendation and language modeling share the same goal: to better predict the next symbol. Understanding this helps us build an overall judgment of model behavior beyond metrics, and make clearer engineering trade-offs among effectiveness, latency, and cost.
