How Heterogeneous Language Models Can Share Symbols Instead of Vectors to Exchange Facts

    Multi-model systems often need to pass intermediate facts between models that handle retrieval, planning, reasoning, code generation, or validation. This article explains why directly sharing hidden states or vectors is difficult across heterogeneous models, reviews the limitations of adapter-based and architecture-constrained approaches, and presents a shared-symbol communication protocol as a practical alternative. It covers symbol granularity, entity and relation indices, context and confidence fields, protocol design, advantages, limitations, and engineering practices for building more flexible cross-model collaboration.

    In a multi-model system, different models can take on different tasks: retrieval, planning, reasoning, code generation, or result validation. One model may extract facts from source material, another may determine relationships between those facts, and a third may generate the final answer. This division of labor can exploit differences in model capability, but it also raises a fundamental question: what exactly should the models exchange?

    The most common answer is natural language. For example, a retrieval model may output “a certain technical approach can reduce communication overhead.” A reasoning model reads that sentence and incorporates it into subsequent judgments. The main advantages of this approach are that it is universal, intuitive, and easy to debug, and it places almost no requirement on participants having the same model structure. However, natural language is designed for human reading. It contains substantial grammatical, formatting, and contextual redundancy. When models need to exchange large amounts of intermediate results at high frequency, a text interface can become a system bottleneck.

    A natural idea is therefore to bypass text and directly exchange hidden states or activations from inside the models. From an information-theoretic perspective, hidden states appear to contain richer semantic information, and the transfer process may be more compact. In practice, however, the problem is far more complex than “sending a set of numbers.”

    1. Why Different Models Cannot Directly Share Vectors

    The hidden states inside a language model are essentially high-dimensional vectors. They do not automatically have cross-model coordinate meanings.

    Suppose Model A processes the concept “red” and forms a vector at a certain layer:

    1. [0.12, -0.48, 1.03, ...]

    Even if Model B also understands “red,” it may use a completely different vector representation:

    1. [-0.77, 0.21, 0.09, ...]

    The two models may differ in several ways:

    1. Vector dimensions are different.
    2. The number of layers and the computation order between layers are different.
    3. Attention mechanisms and normalization methods are different.
    4. Vocabularies, tokenization methods, and training corpora are different.
    5. The same meaning may be rotated, scaled, or distributed differently in the representation space.
    6. The meaning of a vector depends on the context of a specific model rather than existing independently.

    Therefore, the tenth number in a vector does not represent a universal semantic attribute shared across models. Directly feeding Model A’s hidden state into Model B usually causes distribution mismatch. What Model B receives is not a “more condensed fact” but a sequence of abnormal signals that do not match its own training distribution.

    This is similar to why different languages cannot directly share sound waveforms. A sound signal carries information, but the receiver needs matching encoding rules to understand it. If the sender and receiver use different representation systems, the raw signal loses interpretability.

    2. Traditional Solutions and Their Limitations

    Current approaches for passing internal states across models fall roughly into two categories.

    The first category adds an adapter or translator between two models. The sending model produces a hidden state, and the translation module learns to convert it into a representation that the receiving model can process. The form can be abstracted as:

    1. Sending model state hA
    2. ↓
    3. Cross-model mapping module T
    4. ↓
    5. Receiving model state hB

    This approach is relatively direct and can theoretically adapt models with different representation spaces. However, it requires additional training of the mapping module and usually depends on large-scale paired data. When training data is insufficient, the translator may only memorize limited scenarios. If model versions, task types, or contexts change, the mapping quality may also degrade.

    The second category requires collaborating models to have similar structures, such as the same or similar layer layouts, representation dimensions, and training methods. This reduces adaptation difficulty but sacrifices system openness. Real-world multi-model systems often consist of models from different sources, of different sizes, and with different specialties. Requiring architectural consistency limits the freedom of model orchestration.

    These two approaches face the problems of “requiring additional training” and “restricting the range of participants.” More importantly, both attempt to translate one model’s internal coordinates into another model’s internal coordinates instead of rethinking the basic unit of inter-model communication.

    3. Share Symbols, Not Vectors

    A more general approach is for models not to pass hidden vectors directly, but to pass symbol indices that both sides can recognize.

    Here, a “symbol” can be a word, a concept label, an entity identifier, a relation type, or an agreed discrete semantic unit. For example, instead of sending the high-dimensional activation corresponding to “red,” the sending model sends a symbol:

    1. COLOR_RED

    After receiving the symbol, the receiving model uses its own vocabulary representations, concept representations, or internal mappings to restore it to a state suitable for its own computation space.

    The communication flow can be represented as:

    1. Sending model: fact → symbol sequence
    2. Communication protocol: symbol sequence → transmission
    3. Receiving model: symbol sequence → local representation → reasoning

    If natural language is used as the symbol carrier, the message can also be:

    1. ["object:apple", "attribute:color", "value:red"]

    The sender does not need to know the receiver’s vector dimensions, nor does it need to know how a particular layer in the receiver organizes meaning. It only needs to accurately indicate “which concept is being transmitted.” The receiver then reinterprets that concept according to its own representation system.

    This is an important decoupling: the sender determines the semantics, and the receiver performs local encoding. What the two sides share is a symbol system or indexing rule, not hidden-state coordinates.

    4. Why Words or Symbols Can Serve as a Stable Interface

    Certain words or concepts inside a language model often serve more than one task. A concept representing “red” may participate in color recognition, product description, visual reasoning, attribute comparison, and text generation. In other words, vocabulary-related internal representations have some degree of cross-task reusability.

    Of course, this does not mean that a word necessarily has a completely fixed vector. Representations in real models change with context, and the same word may carry different meanings in different sentences. Therefore, symbol communication should not be understood simply as “each word corresponds to a vector that never changes.” More accurately, a symbol provides the receiving model with a stable semantic anchor, and the receiving model then combines context to generate an internal state suitable for the current task.

    For example, the sender transmits:

    1. entity: mercury
    2. property: state
    3. value: liquid
    4. time: room temperature

    The receiver can use its own knowledge and context to convert this set of symbols into reasoning input. Compared with directly passing a vector of unknown origin, this approach is easier to inspect, record, and correct.

    5. How to Design a Symbol Communication Protocol

    For this approach to be truly usable, it is not enough to specify “send words.” A clear communication protocol must also be designed.

    Define Symbol Granularity

    If symbols are too coarse, detail is lost. For example, sending only “apple” does not tell the receiver whether the sender means color, price, or origin. If symbols are too fine, the communication sequence becomes too long and loses its compression advantage.

    A practical approach is to use structured triples or n-tuples:

    1. subject: device A
    2. relation: supports protocol
    3. object: protocol B
    4. confidence: 0.91
    5. source: document snippet

    This structure is more explicit than passing individual words while remaining more concise than full natural language.

    Unify Entity and Relation Indices

    For high-frequency entities, task states, and relation types, stable indices can be assigned. For example:

    1. E17 → a device
    2. R04 → supports
    3. V22 → a protocol

    The indices themselves do not need to carry specific vectors. They only need to remain consistent across both sides’ vocabularies or protocol tables. This reduces transmission length and avoids ambiguity caused by synonyms.

    Preserve Context Scope

    The same symbol may have multiple meanings. Communication messages should include necessary context whenever possible, such as domain, time, conditions, and scope:

    1. concept: temperature increase
    2. condition: ambient pressure constant
    3. scope: experimental phase
    4. conclusion: reaction rate increases

    This prevents the receiving model from incorrectly generalizing a local fact into a universal rule.

    Carry Confidence and Evidence Type

    Symbol communication should not contain only conclusions. It can also include confidence, source category, and verification status:

    1. fact: system supports resumable transfer
    2. confidence: high
    3. status: verified
    4. evidence type: configuration record

    The receiver can use these fields to decide whether to use the information directly or request further verification from another model.

    6. Advantages Compared with Natural-Language Communication

    First, communication content is more compact. Natural language must include grammar and explanatory elements, while a symbol protocol retains only the semantic units required by the task, making it suitable for high-frequency, multi-turn model collaboration.

    Second, the receiver can perform local reconstruction. The sender does not need to understand the receiving model’s architecture or convert its own hidden states into the other model’s vector format.

    Third, interpretability is stronger. Vectors are difficult to review manually, while symbol messages can be recorded, retrieved, replayed, and audited. When errors occur, engineers can determine whether the problem is fact extraction, symbol mapping, or reasoning on the receiving side.

    Fourth, the system is easier to extend. A new model can join the collaboration network as long as it supports the common symbol protocol. There is no need to train a separate vector translator for every pair of models.

    Fifth, security control becomes easier. The system can validate symbol sets, relation types, and field ranges, preventing unauthorized instructions or abnormal data from entering downstream reasoning processes.

    7. Limitations to Acknowledge

    Symbol communication is not without cost.

    First, discrete symbols may lose continuous information. Some complex emotions, probability distributions, visual details, or long-range reasoning processes are difficult to express completely with a small number of words.

    Second, the symbol system requires governance. If different models interpret the same label differently, communication will still produce semantic conflicts. The protocol should therefore include versions, definitions, and compatibility strategies.

    Third, local reconstruction on the receiving side may introduce bias. The same symbol may have different levels of knowledge coverage in different models, and the receiving model may be unable to recover all the background that the sender intended to express.

    Finally, fact transmission and reasoning transmission need to be distinguished. A model can transmit “an observed fact” or “a conclusion based on facts.” The two have different levels of credibility. The protocol should ideally label the reasoning level explicitly to avoid treating unverified judgments as raw facts.

    8. Engineering Implementation Recommendations

    In a practical system, symbol communication can be designed as a layered structure:

    1. Layer 1: entities and concepts
    2. Layer 2: attributes, relations, and states
    3. Layer 3: conditions, time, and scope
    4. Layer 4: confidence, evidence, and verification results
    5. Layer 5: task control information

    A simple example:

    1. {
    2. "entity": "E17",
    3. "relation": "R04",
    4. "value": "V22",
    5. "condition": "room temperature",
    6. "confidence": 0.91,
    7. "verification": "verified"
    8. }

    Messages of this kind can be generated by models or validated by rules. After reading them, the receiving model converts them into prompts, concept activations, or internal memory suitable for its own context.

    In practice, the following mechanisms should also be added:

    • Protocol version control to prevent arbitrary changes in symbol meanings.
    • Fallback for unknown symbols, passing unrecognized indices to a text explanation module.
    • Conflict detection, pausing automatic merging when multiple models give different values for the same fact.
    • Confidence propagation, preventing low-credibility information from being repeatedly amplified across a multi-model chain.
    • Traceable logs, recording symbol sources, conversion processes, and final usage locations.
    • Human-in-the-loop review, requiring additional confirmation for high-risk facts.

    9. Conclusion

    What different language models truly need to share is usually not each other’s internal coordinates, but semantic units that can be reinterpreted. Directly exchanging vectors depends on implicit representation alignment between models. Exchanging natural language can introduce redundancy and latency. Shared symbol indices offer another compromise: the sender outputs symbols corresponding to facts, and the receiver uses its own representation space to perform recovery and reasoning.

    The core value of this approach is not eliminating all information loss. Rather, it reframes cross-model communication from a “coordinate transformation problem” into a “semantic protocol problem.” As long as symbol definitions are clear, context is complete, and confidence is traceable, heterogeneous models can establish more flexible collaboration without sharing vectors or forcing architectural consistency. For future multi-model orchestration systems, this symbol-oriented communication approach has the potential to become an important foundation for connecting retrieval, reasoning, planning, and execution modules.