Why System Prompts Keep Growing: Size Estimates and Ways to Trim Them

1. Why prompts keep getting longer

System prompts in large-model applications are usually built by concatenating several modules. They often start with a role and a tone, then gradually add memory read/write rules, tool use, format conventions, and safety boundaries. Each new capability needs text that tells the model what it can do, when to do it, and how. The model does not know about external features on its own, so those instructions have to enter the context.

That growth can be justified, but its speed and scale should be measured with data, not judged by impression.

2. Define what “bloat” is compared against

Before discussing prompt length, settle three things.

First, the entry point. Web or client chat products aimed at general users usually attach fairly complete behavior rules and feature descriptions. When calling an API directly, developers choose the system message themselves, and the platform injects less by default. Dedicated coding assistants use another set of instructions for editing, terminals, and version control. These three should not be compared directly.

Second, feature flags. In the same product, turning off web retrieval, memory, and tools keeps the related rules out of context, so total length drops sharply. Full configuration and plain chat should be counted separately.

Third, a version baseline. To judge whether an update caused bloat, compare character and token counts of two versions under the same entry point and the same flags. Without that comparison, you can only say the content increased, not by how much.

3. Converting characters to tokens

A token is the unit models use to process text, and it does not map one-to-one to characters. English words, code, XML tags, and JSON differ in token density. For mixed English, markup, and code, a rough rule is about 3 to 4 characters per token. A text of 200,000 characters is then roughly 50,000 to 70,000 tokens. Chinese conversion varies more; a common rule of thumb is about 1 to 2 Chinese characters per token, depending on tokenization.

A rough budget can use pseudocode like this:

function estimate_tokens(text, ratio):
    # ratio: average characters per token; about 3.5 for mixed English text
    return ceil(length(text) / ratio)
budget = estimate_tokens(system_prompt, 3.5)
if budget > context_limit * 0.15:
    warn("system prompt share of context is high")

This estimate is only for budgeting and alerts. Actual figures should come from usage reported by the service.

4. What a system prompt usually contains

A complete system prompt typically has five kinds of content.

4.1 Base behavior rules

This section defines how the model speaks and where it stops: tone, answer length, refusal scope, safety rules, how to disclaim legal and financial questions, how to handle mental-health topics, neutrality on contested topics, and the training-data cutoff. These rules matter for product compliance and usually cannot be removed, but similar items can be merged so the same requirement is not repeated in several places.

4.2 Runtime environment

This includes the current date, model version, product form, language preference, and when the latest information should be retrieved. Date and version change with deployment, so they should be inserted by the program at request time rather than hardcoded into a long static text.

4.3 Memory instructions

If the system supports memory across sessions, the prompt needs to explain how memory is organized, when it is read, and how it is written. A common approach is to split memory into files such as a user profile, preferences, topic notes, domain notes, and people. The instructions should also state what must not be stored, such as sensitive credentials, other people’s private data, and temporary information.

A typical layout looks like this:

/memory
  /profile.md        # basic user information, read when needed
  /preferences.md    # output preferences, read at the start of each session
  /topics/           # notes on specific topics
  /areas/            # long-term areas of interest
  /people/           # people related to the user; only necessary details

4.4 Tool definitions

Tool definitions are among the fastest-growing parts of the prompt. Each tool needs a name, a description, and a parameter structure, often as JSON Schema. The model can call only tools it has seen in context, so these definitions usually have to be sent with the request.

4.5 Output and interaction conventions

These cover format requirements, code-block rules, how visual output is produced, and definitions of interactive controls when the user must choose. This material is relatively stable and fits a fixed template.

5. The cost and design of tool definitions

Tool-definition tokens count as request input. A typical search tool looks like this:

{
  "name": "web_search",
  "description": "Call when the question involves recent information or the user explicitly asks for a search.",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "description": "Search keywords" },
      "max_results": { "type": "integer", "minimum": 1, "maximum": 10 }
    },
    "required": ["query"]
  }
}

The more tools there are, the larger the fixed cost of every request. Several ways reduce that cost.

1. Load on demand. Inject only the tools relevant to the task. Plain text editing does not need search or file operations.
2. Merge similar tools. Operations with similar parameters can become one tool distinguished by an action field. Merging too far makes parameters harder to fill correctly, so the tradeoff has to be weighed.
3. Shorten descriptions. A description should say when to call the tool, not restate implementation details.

6. How to read and write memory files

The hard part of a memory system is controlling how much is read. Loading every memory file at the start of each session inflates context quickly, and most of it is irrelevant to the current question. A more practical pattern is:

• At session start, read only a small set of necessary files, such as preferences.
• When a topic comes up, read the matching topics file.
• Keep written entries short, and periodically merge duplicates.

Memory writes should also have clear filters. Credentials, ID numbers, medical details, and similar sensitive information should not be persisted unless the product truly needs them and authorization is in place.

7. Move “write it in the prompt” into code

Many prompts grow because they do work the program should do. For example:

• Date math, unit conversion, and format checks can live in code instead of long prompt rules.
• Permission checks should run on the server; the prompt can only assist.
• For fixed output shapes, structured-output constraints or post-processing can replace lengthy format descriptions.

Rules in code can be tested, and changing them is less risky.

8. Baselines and regression checks for length

The prompt is part of the system and should be versioned. Useful mechanisms include:

1. Record character and token counts for each version under different entry points and feature flags.
2. Compare each change with the baseline, and require a reason when a threshold is exceeded.
3. Keep a fixed set of test questions to check that behavior is still correct after rules are removed.
4. Put the stable prefix at the start of the request and dynamic content at the end, so the service can reuse a cache. Actual savings from prefix caching depend on the provider and should be measured.

9. A practical checklist

• Compare length only under the same entry point, the same feature flags, and a clear version baseline.
• Estimate tokens for budgeting, and treat server usage stats as the source of truth.
• Keep compliance rules, but merge duplicates; inject date and version dynamically.
• Load tools and memory files on demand, and keep descriptions and stored entries short.
• Move calculable, permission, and format rules into code or structured constraints where possible.
• Version the prompt, alert on growth past a threshold, and retest behavior after cuts.