# Shared Symbols, Not Shared Vectors: Letting Language Models Pass Facts to Each Other

    Multi-model collaboration typically relies on natural language as the communication medium, which causes information loss, extra encoding overhead, and bloated context windows. Directly sharing hidden states fails because different models have incompatible representation spaces. This article explains a training-free, architecture-agnostic alternative: instead of transmitting vectors, collaborating models exchange symbol indices tied to a shared vocabulary, with each model maintaining its own symbol-to-vector lookup table. It covers how the protocol works, why it sidesteps both translation training and structural constraints, practical engineering considerations, and its limitations.

    1. The Dilemma of Multi-Model Collaboration

    Having multiple language models divide up work is a common architecture for building complex AI systems today: one model handles retrieval and research, another handles logical reasoning, another generates code—each playing to its strengths and assembling into a more capable whole.

    In such systems, how do models exchange information? The mainstream answer today is natural language—one model writes its conclusions as text, and the other model reads and interprets that text. This “passing notes in plain language” approach is maximally compatible: any model can read text. But the costs are clear:

    • Information loss: A model’s internal state is a highly dense continuous vector. Compressing it into discrete text throws away a great deal of detail.
    • Round-trip overhead: The receiving model must re-encode the text back into its own internal representation, adding two extra rounds of computation.
    • Verbose expression: Many complex facts can’t be summarized in a few sentences. The text grows longer, and the context window fills up quickly.

    A natural question follows: can we skip the text entirely and hand model A’s internal states (hidden states / activations) directly to model B?

    2. Why Directly Passing Vectors Doesn’t Work

    The answer: each model’s “brain wiring” is fundamentally incompatible.

    During pretraining, every model independently learns its own representation space. The concept of “red” might be a particular 768-dimensional vector in model A’s latent space, and a completely different 4096-dimensional vector in model B’s—not only are the values unrelated, even the dimensions don’t match. Feeding A’s activations directly into B just looks like meaningless noise to B; you’d be better off passing text.

    The research community has mainly pursued two routes:

    1. Train a translation module: Train a dedicated mapping network that learns to “translate” model A’s internal states into a form model B can understand. The cost is extra training data and compute—and whenever you swap in a different model pair, the translator must be retrained.
    2. Restrict to isomorphic models: Require collaborating models to be structurally similar with sufficiently close representation spaces, so activations can be exchanged directly or approximately. The cost is a drastic reduction in the range of models that can be combined—models with different architectures or scales can’t team up.

    Both routes have serious drawbacks: the first introduces training costs, the second restricts compositional freedom. Is there a third path that is both training-free and free of structural constraints?

    3. The Third Path: Share Symbols, Not Vectors

    Research on language model internal representations offers a key insight: within a single model, a specific word (say, “red”) tends to correspond to a relatively stable set of internal representations, and that set can support the model’s reasoning across many different kinds of questions—it isn’t tied to one particular task.

    This leads to an elegant design: since every model can, internally, find its own vector representation for the word “red,” why not have collaborating models each maintain their own “word-to-vector” lookup table around the same set of words?

    The idea in one sentence: don’t transmit vectors—transmit words.

    3.1 How the Communication Protocol Works

    The carrier of communication is no longer dense vectors, but a set of symbols (vocabulary) agreed upon in advance by both sides. The full flow:

    1. Build the tables: The sender and receiver agree on a shared symbol set (drawn from a general vocabulary or customized to the task domain). Each model retrieves the hidden states corresponding to these symbols internally, forming its own “symbol → vector” index.
    2. Send: When model A wants to convey a fact, it doesn’t output raw vectors. Instead, it retrieves “which symbols’ internal representations are closest to what I want to convey” and sends out that small batch of symbol indices.
    3. Receive: After model B receives the symbol indices, it looks them up in its own table, pulls out these symbols’ vectors in B’s own latent space, and injects them into its internal state for subsequent reasoning.
    4. Return: The same process applies in the reverse direction.

    Illustrative pseudocode:

    1. # Initialization: each side builds its own table (one-time, training-free)
    2. symbol_vocab = ["red", "high temperature", "protocol", "growth rate", ...] # shared symbol set
    3. table_A = {sym: model_A.hidden_state_of(sym) for sym in symbol_vocab}
    4. table_B = {sym: model_B.hidden_state_of(sym) for sym in symbol_vocab}
    5. # Communication: A → B
    6. msg = model_A.encode_fact_as_symbols(fact, table_A)
    7. # msg looks like ["growth rate", "high temperature"] — only symbol indices are sent, no vectors
    8. # Decoding on B's side: look up B's own table, convert to B's internal representations
    9. latents = [table_B[sym] for sym in msg]
    10. model_B.inject(latents) # inject into hidden states, directly participating in reasoning

    3.2 Why This Sidesteps Both Obstacles

    The key is that it downgrades the object of “alignment” from the vector space to the symbol space:

    • No trained translator needed: The “translation” happens at the table-lookup step, and each model maintains its own table—no cross-model mapping network needs to be learned. In essence, vocabulary serves as a common coordinate system between the two latent spaces.
    • No requirement for isomorphic models: The symbol table is a purely semantic-level agreement, fully decoupled from model architecture, parameter scale, and hidden-layer dimensions. A large model and a small model can communicate as long as both can produce internal representations for the same set of words.

    An analogy: two people who don’t speak each other’s language don’t need to learn each other’s pronunciation (vectors). As long as each holds a dictionary organized with the same page numbers (symbol tables), they can communicate through ultra-lightweight information like “turn to page N.”

    3.3 The Essential Difference from Natural-Language Interaction

    Note that transmitting symbols is not the same as transmitting text. The difference lies in the “semantic density” of what gets transmitted:

    • When transmitting text, the receiver must perform full encoding from scratch, gradually accumulating discrete tokens into a semantic representation.
    • When transmitting symbol indices, the receiver directly injects its own mature internal representation of each symbol as a seed into the reasoning process, skipping the “text-to-semantics” cold start.

    Symbol communication is therefore closer to a form of semantic-level compression: with a tiny index overhead, it activates an information-dense state inside the receiver.

    4. Engineering Considerations for Real-World Deployment

    Applying this approach to real multi-model systems raises several practical issues:

    1. Symbol table design: The symbol set must cover the key concepts of the task domain. General tasks can use a general vocabulary; vertical domains (such as healthcare or law) should incorporate domain terminology—otherwise the sender can’t find suitable symbols, and information gets distorted.
    2. Representation stability: The same symbol’s internal representation shifts across different contexts. In practice, you typically aggregate the symbol’s representations across several typical contexts to obtain a more robust anchor vector.
    3. Ambiguity in multi-symbol combinations: A single symbol has clear semantics, but a string of symbols combined may be ambiguous. You can transmit in order, attach positional indices, or superimpose relative position information when injecting into the receiver.
    4. Consistency and auditing: Symbol communication lacks the readability and auditability of text. During debugging, you can keep a bypass log mapping symbols to text, making it easier to investigate “what exactly did the models pass to each other.”
    5. Infrastructure support: For such systems, the symbol tables, communication logs, and orchestration state are best managed in a dedicated service layer—for example, using a message queue to carry the symbol message stream between models, a vector retrieval service to maintain large-scale symbol indices, and a logging service to retain communication traces for replay and evaluation.

    5. Limitations and Outlook

    This route is not a silver bullet. First, the granularity of transmitted information is bounded by the symbol table’s expressiveness—for complex structures that are hard to summarize with discrete words (such as a complete proof of a reasoning step), symbol indices may not fully cover them. Second, symbol retrieval depends on the quality of each model’s internal representations; if a model’s understanding of a concept is itself flawed, communication will amplify that flaw. Additionally, multiple models may disagree on the semantic boundaries of the same symbol—whether “red” refers to a hue or a metaphor still requires task-level constraints as a safeguard.

    But its value lies in drawing a clear design boundary: interoperability between models doesn’t have to be built on aligned representation spaces—it can be built on symbol-level agreements. This offers a middle path for heterogeneous model collaboration with zero training cost and high compositional freedom. As multi-agent systems scale up, how to further extend the dynamism of symbol tables and improve the precision of compositional semantics will be a question worth continued exploration in this direction.