LLM Content Moderation: Why Individually Harmless Inputs Become Toxic in Combination
This article examines a structural weakness in LLM content moderation: individually harmless inputs can combine into a harmful attack without triggering any single-item alert. It explains why traditional single-input moderation fails against multi-turn splitting, cross-carrier assembly, and tool-chain chaining, then breaks down five types of mismatch between moderation and attacks. The core proposal is to reframe moderation from single-point interception into stateful, session-level risk governance, backed by a layered defense architecture, a session-level risk accumulation model, context-aware moderation, and governance for retrieved content, memory, and tool calls. It also gives scenario-specific guidance for coding assistants, AI agents, and public-facing services, covers evaluation using compositional attack sets, and shows how to assemble an operable moderation pipeline from general cloud services.
Introduction: All Compliance Checks Passed, Yet the Dangerous Request Was Completed
Imagine a session with an AI coding assistant:
- The user asks the assistant to implement an input validation function. The module logic is harmless, and the validation passes.
- The user then asks the assistant to “keep the compatibility entry that legacy callers depend on.” On its own, this is a normal compatibility refactor, and it passes again.
- The user asks the assistant to “organize the call chain of the step 2 changes into a change description” for review submission. A change description is a routine deliverable, so it passes again.
Each of the three inputs goes through content moderation independently, and each result is “harmless.” Yet combined, they complete a targeted weakening of authentication checks, and no single-item alert fires along the way. This is the compositional mismatch in LLM content moderation: the unit of judgment is a single input, while the unit of risk generation is a group of inputs together with their context.
This kind of mismatch is not an occasional model misjudgment; it is a structural flaw at the level of the moderation architecture. As long as moderation keeps working in a “single, static, input-only” way, multi-turn splitting, cross-carrier assembly, and tool-chain chaining can bypass it. Starting from the failure mechanism, this article discusses how to restructure moderation from “single-point interception” into “stateful, session-level risk governance.”
1. What Is Compositional Mismatch
Classic content moderation is mostly designed for user-generated content scenarios. Its basic assumption is that risk is carried within a single piece of content, and that vocabulary, phrasing, or semantics can directly expose violating intent. Under this assumption, “judge item by item, intercept on discovery” is sufficient.
LLM conversation breaks this assumption:
- The interaction unit is the session, not the single message. A complete intent often emerges gradually across multiple rounds of input.
- The context window is itself an attack surface. Historical messages, retrieved documents, and tool-returned results all enter the model’s field of judgment, but their trustworthiness varies.
- Agents turn “saying” into “doing.” The harm from violating content can be caught by intercepting the output, but a single unauthorized agent operation can land directly on a real system.
Once these three effects stack, the limits of single-item moderation are fully exposed. Risks can be grouped by combination dimension into four categories:
| Combination dimension | Typical form | Blind spot of single-item moderation |
|---|---|---|
| Multi-turn combination | Step-by-step extraction: ask about the principle first, then parameters, then assemble a complete dangerous plan | Each round looks like a “general knowledge Q&A” |
| Cross-carrier combination | Provide a compliant outline in the conversation, then fill in violating details in retrieved documents or attachments | Conversation moderation and retrieved-content moderation each cover only one segment |
| Tool combination | Each single-step operation is within authorization, but chained together they form a complete attack chain or data exfiltration path | Only single-call parameters are checked; the overall effect of the task chain is not assessed |
| Long-term combination | Memory or preferences accumulated across sessions and versions are gradually poisoned | Both memory write and read lack a risk perspective |
2. Failure Mechanisms: Five Mismatches Between Moderation and Attack
1. Granularity mismatch: the moderation object and the risk unit are misaligned
Moderation answers the question “is this one item violating.” What the attacker wants to do is “make every item compliant, and make them violating when combined.” The simplest way to reduce the harmful appearance of a single round is to split:
- Split by step: cut a complete plan into four segments — principle, parameters, implementation, optimization — and ask about them one by one.
- Split by carrier: put different fragments into the conversation, files, links, and tool parameters separately.
- Split by time: stretch the span over several days and use the memory mechanism to accumulate across sessions.
Each fragment stands up to scrutiny on its own; only when assembled does it become the complete intent. If moderation does no cross-item correlation, this kind of attack is almost undetectable.
2. Symbol mismatch: surface features are disconnected from real semantics
Mainstream content moderation models identify direct requests reasonably well under “single, context-free” conditions, but once the input is rewritten, the recognition rate drops noticeably. Common rewriting techniques include:
- Semantic equivalent substitution: replacing sensitive objects with code names, metaphors, or fictional settings;
- Structured packaging: rewriting into configuration files, pseudo-code, unit tests, or tables;
- Language and cultural conversion: using minority languages, dialects, or expressions that need “translation before understanding”;
- Task packaging: requesting details under the name of “writing material,” “technical demo,” or “security research.”
If moderation only looks at surface features without restoring the intent, these transformations amount to a natural pass.
3. Timing and stage mismatch: the moderation position is disconnected from the point where risk occurs
In the common pipeline of “moderate the input before generation, moderate the output after generation,” the positioning looks complete, but it misses a key link: the evolution of session state. The third-round input looks fine on its own, but the first two rounds have already pushed the topic to a sensitive boundary. Without state, moderation knows nothing about “which round it is” or “what was said before.”
Stage mismatch is just as common. Many teams only defend at the request entrance, while overlooking:
- Memory writes: long-term memory and user profiles, once written, continuously affect all subsequent sessions;
- Tool-output backflow: web pages, files, and retrieval results enter the context without review and may carry embedded instructions;
- Session summaries: compressed summaries generated to save context may retain key details while dropping risk warnings.
4. Identity and channel mismatch: moderation is treated as a data problem
Many systems equate content moderation with a “model problem” and hand it to a single classification model, yet in practice the mismatch often happens in the process:
- Different channels (web, open API, agents, embedded components) connect moderation policies of different strengths, forming “weakest-link entrances”;
- Moderation is provided segment by segment by multiple vendors, each covering its own part, with combined risk left unbacked;
- Security policy updates require cross-team releases, so newly appeared attack methods go uncovered for weeks.
5. Behavior and semantics mismatch: output-level moderation cannot control the execution layer
Agents turn the model from an “answerer” into an “executor,” and the center of risk shifts accordingly:
- Outputting an ordinary-looking call instruction may trigger a real deletion, transfer, or permission change;
- Sensitive data flowing between tools usually does not pass through content moderation and shows up only as ordinary field parameters;
- A seemingly harmless automation flow, once invoked repeatedly, can produce denial-of-service-style resource consumption.
Auditing only “what was said” and not “what was done” is, in agent scenarios, equivalent to having no defense.
3. Protection Reconstruction: From Single-Point Moderation to Session-Level Risk Governance
3.1 Principles First
- Stateful moderation: take the session, not the message, as the basic unit.
- Judge by accumulation: focus on the additive effect of risk, not just the appearance of a single round.
- Re-audit the output: use the session summary as moderation context and do a second confirmation on generated content.
- Constrain behavior: use permissions, sandboxes, and explicit confirmation to backstop model misjudgment.
- Evaluate the policy: before any rule goes live, validate its effect on a compositional attack set.
3.2 Layered Defense Architecture
| Layer | Main means | Problem it solves |
|---|---|---|
| L1 Entrance lightweight filter | Rules and keywords, input normalization, lightweight classification model | Low-cost interception of explicit violations, and triage for deeper moderation |
| L2 Session state layer | Session summary, intent labels, risk score, time decay | Restore multi-turn accumulation and mark topic escalation |
| L3 Context moderation | Re-judge while carrying the session summary and the most recent rounds | Identify “harmless alone, harmful together” |
| L4 Retrieval and tool content governance | Source grading, instruction isolation, output backflow moderation | Prevent external content from carrying instructions |
| L5 Behavior and permission layer | Least privilege, explicit confirmation of sensitive operations, sandbox execution, rate limiting and quota | Backstop model misjudgment and limit actual harm |
| L6 Output moderation and audit | Pre-generation recheck, response template constraints, full-chain logging | Reduce harmful output and support post-hoc tracing |
The key to the layered design is: any single layer can fail, but not all layers should fail at the same time. L1 pursues low cost and low latency, L3 pursues accuracy and context understanding, and L5 does not depend on model judgment. They solve different problems and should not replace each other.
3.3 Session State and Risk Accumulation Model
Session state should contain at least three parts:
- Session summary: a structured compression of the dialogue history that preserves the task goal, involved objects, completed steps, and pending questions;
- Intent labels: such as “knowledge Q&A,” “code refactoring,” “data export,” and “system configuration”;
- Risk trajectory: each round’s risk score, its direction of change, and the trigger reason.
The risk score should not use simple aggregation like “sum” or “take the maximum.” Summing makes long sessions naturally lean toward high risk, and the false-positive rate in long-task scenarios rises sharply; taking the maximum, in turn, discards accumulation information. A more reasonable approach is weighted accumulation with time decay, so that recent high-risk inputs receive higher weight:
# Illustrative code: session-level risk accumulation (pseudocode, not a specific product API)from dataclasses import dataclass, field@dataclassclass TurnRisk:score: float # single-round base risk score, taken from L1/L2 moderation resultsintent: str # intent labeltopics: list # objects or topics involvedcontext_risk: float = 0.0 # context risk score after combining session history@dataclassclass SessionState:turns: list = field(default_factory=list)# escalate level by level; specific values need scenario calibrationthresholds: tuple = (0.30, 0.55, 0.75)def accumulate(state: SessionState, turn: TurnRisk, decay: float = 0.8) -> float:"""Weighted accumulation + decay: recent behavior carries higher weight;single-round overstepping is permanently counted as high-risk evidence."""prior = sum(t.context_risk * (decay ** i) for i, t in enumerate(reversed(state.turns)))risk = max(prior, turn.score, turn.context_risk)if turn.intent in HIGH_RISK_INTENTS: # e.g. data export, permission changerisk = max(risk, state.thresholds[-1])return min(risk, 1.0)def decide(risk: float, state: SessionState) -> str:low, mid, high = state.thresholdsif risk >= high:return "block_or_human_review" # block and forward to human reviewif risk >= mid:return "recheck_with_context" # re-moderate with contextif risk >= low:return "watermark_and_log" # mark the session and log itreturn "pass"
There are four key points in designing this model:
- Decay: each round’s prior risk contribution is multiplied by a decay coefficient, preventing historical risk from accumulating indefinitely;
- Intent elevation: some intents change the nature of the harm once they stack. For example, knowledge Q&A combined with data export can shift from “science popularization” to “extracting sensitive information from memory,” and such combinations should be elevated directly into the high-risk range;
- Threshold differentiation: scenarios like code generation and operations work inherently require high privilege, so they should not reuse the security baseline meant for public-facing services, and thresholds should be set per business line;
- State isolation: the risk trajectory serves only security judgment and has a defined retention period; it should not be mixed with business profiles.
3.4 Context Moderation: Let Moderation “See” What Happened Before
The core of context moderation is to reconstruct the unit of judgment. What is fed to the moderation model on each pass is not only the current message, but also the session summary, the original text of the most recent rounds, and the current intent label. A simplified scheduling logic:
def should_recheck(state, turn) -> bool:if turn.score >= 0.2: # moderate risk: recheck the next round with contextreturn Trueif turn.intent != state.last_intent and turn.score >= 0.1: # topic switchreturn Trueif state.memory_updated: # this round wrote long-term memoryreturn Truereturn Falsedef recheck(state, text):context = build_context(state) # summary + recent N rounds + intent labelverdict = audit_model(text, context) # output: risk level + basis + combination relationif verdict.level == "high" and verdict.composed_with:# high risk and composed across multiple rounds: block and roll back the risky roundsreturn "block", verdictreturn "pass", verdict
Together with context moderation, two engineering measures are needed:
- Topic escalation detection: identify a progressive trajectory of “normal question → edge question → specific detail.” The judgment basis is the rate of change of topic sensitivity, not the absolute value at a single point.
- Combination tracing: the moderation result needs to explain which rounds and which carriers jointly constitute the risk. This both gives human reviewers a basis and lets the strategy team induct new rules from individual cases.
3.5 Governance of Retrieval, Memory, and Tool Content
The risk entrances for LLMs are not only user input. Retrieved web pages, uploaded documents, historical entries in a memory store, and JSON returned by tools can all carry instructions aimed at the model. Governance should follow three principles:
- Source grading: the trust level decreases in order across user input, the in-house knowledge base, external web pages, and data returned by third-party interfaces;
- Content-instruction isolation: external content is uniformly presented as “data,” with its source and trustworthiness explicitly labeled; for outputs involving external content, guide the model to state its basis so it can be verified afterward;
- Write review: writes to long-term memory and user profiles must pass a risk judgment. Record the source and purpose at write time, and re-evaluate relevance against the current task at read time, rather than treating memory as permanently trusted context.
At the tool-call level, it is advisable to introduce task-chain evaluation: in the planning stage, an independent policy checks the entire execution chain, including each step’s goal, the source of its parameters, and its final impact. Operations involving deletion, transfer, or permission changes must go through explicit confirmation.
4. Scenario-Based Protection
4.1 The Intelligent Programming Scenario: Beware the “Legitimate Refactoring” Shell
Compositional risk in a programming assistant is especially hidden, because violating code often wears a legitimate shell: variables are abstracted, functions are split, and security controls are labeled as “kept for compatibility.” Beyond general moderation, three additional types of checks are recommended:
- Code intent moderation: combine with the session summary to judge the cumulative effect of this change together with past changes. For example, if earlier sessions had raised multiple requests around “caller compatibility” and this time the request is to remove an authentication check, it should be treated as a compositional risk signal.
- Change description review: the commit descriptions and change summaries produced by a code assistant flow along with the code, so they need to be checked for whether they truthfully describe the intent of the change, guarding against “compliant description with overstepping code” or “compliant code with a distorted description.”
- Secret and personal information leak detection: scanning rules and desensitization capabilities are relatively mature. In ultra-large-scale repository write scenarios, the check must run in parallel with the write flow and cannot become a throughput bottleneck.
Engineering can also borrow the mature compositional testing ideas from software engineering: migrate the methods of functional testing, mutation testing, and property testing into the security domain, and verify system behavior by combination rather than by single points.
4.2 Agent Automation: From Content Moderation to Behavior Moderation
Personal agents take on tasks such as schedule management, file processing, message sending, and code execution. A single operation carries limited risk, but the compositional risk is extremely high. It is advisable to build constraints along these lines:
- Capability minimization: grant tool permissions per task, and by default do not open sensitive capabilities such as deletion, payment, and external sending;
- Graded confirmation: classify operations by “impact scope × reversibility,” and require explicit user confirmation for high-impact, irreversible operations;
- Resource quota: set limits on call count, time, and cost for loop and batch tasks, to prevent runaway loops from exhausting resources;
- Sandbox execution: run code and scripts in an isolated environment, with restricted filesystem and network access;
- Full-chain audit: associate each round’s input, tool call, returned result, and final action, so the task chain can be reconstructed afterward.
4.3 Public-Facing Services: Control the Split Space and Cross-Carrier Channels
For public model services, the focus of protection is to compress the attacker’s space for splitting:
- Enable session-level risk tracking by default on the open-API side;
- Apply equal-strength moderation to multimodal carriers such as uploaded files and image attachments, to avoid “clean text with a payload hidden in the attachment”;
- For external links in retrieval-augmented scenarios, restrict the range they can be cited from, and perform content summarization and instruction filtering when necessary.
5. Evaluation: Test Whether Protection Really Works Using Compositional Attack Sets
Measuring moderation effectiveness with only single-round samples will systematically overestimate the protection capability. It is advisable to build a layered compositional attack evaluation set:
| Layer | Composition method | Test focus |
|---|---|---|
| Single-round baseline | Existing violation samples + normal business samples | Basic recall rate and false-positive rate |
| Multi-round combination | Split a complete violating intent step by step | Whether risk accumulation and topic escalation detection take effect |
| Cross-carrier combination | Violation details scattered across the conversation, files, and links | The coordination ability of each carrier’s moderation |
| Tool-chain combination | Single steps compliant, overstepping when chained | Whether task-chain evaluation and behavior constraints are effective |
| Normal long business session | Real long-task conversations | Whether the false-positive rate rises significantly with session length |
Metric design should cover both security and experience:
- Compositional Attack Success Rate (CASR): the proportion of attacks where the attacker completes the goal using only information already present in the session;
- Harm accumulation coefficient: the proportion of violating output that comes from beyond the current input;
- Average breakthrough rounds: the number of rounds needed from the start of the session to the first violating output;
- Session-level false-positive rate: the proportion of normal sessions that get intercepted;
- Moderation incremental latency: the extra response time introduced by moderation, which needs a budget set per business scenario, for example keeping the incremental latency in interactive dialogue on the order of hundreds of milliseconds.
The last point is especially important. If moderation slows down the normal experience, business teams often choose “temporary degradation” under high concurrency, and the protection system ends up failing precisely when it is needed most. The evaluation should also add a degradation test: simulate the moderation service being unavailable, and check whether the system can automatically switch to a more conservative policy (such as limiting tool permissions or forcing confirmation) rather than releasing without protection.
6. Cloud Implementation: Assemble an Operable Moderation Pipeline from General Capabilities
The architecture above does not need to be built entirely from scratch; many capabilities can be composed as general cloud services:
- Content security moderation: handles the model judgment for L1 and L3;
- Message queue: asynchronously routes high-latency moderation to avoid blocking the main path;
- Object storage: centrally stores audit logs and attack samples;
- Log service and monitoring alerts: track the distribution of risk scores and changes in CASR and false-positive rate, and trigger alerts on abnormal fluctuations;
- Container platform: provides an isolated sandbox for executing code and scripts;
- Function compute: handles lightweight rule filtering and rate-limiting logic.
The content security moderation products from Baidu AI Cloud belong to this general category and can serve as a reference for technical selection. For the specific access method, interface definitions, and billing rules, refer to the official documentation and actual testing. During implementation, first map the business process, clarify which stages require synchronous moderation and which can be processed asynchronously, and then allocate moderation resources according to the latency budget.
Conclusion
“Harmless individually, toxic in combination” exposes not a flaw in some particular moderation model, but a mismatch between the moderation paradigm and the form of risk. Once the unit of risk shifts from “a single piece of content” to “a session, a task chain, a set of tool calls,” moderation must have a stateful, context-aware, and behavior-aware perspective.
The pragmatic path is to use low-cost lightweight filtering to cover the vast majority of explicit violations, concentrate compute on context moderation and compositional analysis, and then use permissions and sandboxes to backstop model misjudgment. Finally, use a compositional attack evaluation set to calibrate continuously. Only when protection can be measured, regression-tested, and continuously updated does it truly become a system capability, rather than a one-time configuration at launch.
