Shared Symbols, Not Vectors: Efficient Fact Transfer Between Heterogeneous Models

    Multi-model collaboration systems usually let models exchange intermediate results as natural language, which is both lossy and expensive, while passing raw hidden states directly fails because different models have no shared metric for their internal spaces. This article presents an alternative: align on the shared vocabulary layer instead of the vector layer. Each model keeps its own symbol-to-representation table, models exchange only discrete token symbols, and the receiver decodes those symbols into its own internal space. The article covers the prerequisites (verifying vocabulary overlap, token coverage, OOV rate and tokenizer compatibility), a reference encode/decode implementation, a comparison against natural language and trained mapper approaches, the boundaries and failure modes with fallback strategies, and a practical deployment checklist.

    1. When models need to work as a team

    In multi-model collaboration systems, splitting work across specialized models has become a common pattern. One model is good at retrieval and fact-checking, another at logical reasoning, and a third at code generation or text polishing. Each contributes its strength, and together they deliver more capability than any single model.

    But collaboration depends on one precondition: the models must be able to talk to each other. In most systems, models communicate through natural language — intermediate results are serialized sentence by sentence and handed downstream. This kind of heterogeneous model communication is not wrong, but it carries two unavoidable costs:

    • Text is lossy and verbose. The high-dimensional state a model computed internally collapses into a limited number of tokens, and much of the information is lost along the way.
    • Serialization and re-parsing overhead stack up across multiple round trips, and both latency and cost stay high.

    So a natural question arises: can we skip text and exchange what the models have “in mind” — their internal hidden states (activations) directly? That is the core problem of efficient fact transfer between heterogeneous models.

    2. Why passing raw vectors does not work

    The difficulty is that different models do not share an aligned internal coordinate system.

    When model A represents the concept “red,” that concept maps to a specific set of dimensions. When model B represents the same “red,” it may map to an entirely different set of values — not necessarily the same dimensionality, and with no correspondence in value distribution. Handing A’s raw activations to B leaves B interpreting a block of meaningless noise. It is like playing a Chinese audio waveform to someone who only understands English: the signal arrives intact, but the meaning is gone.

    The conclusion in one sentence: vector-level communication lacks a metric shared by both models, and that is the root reason direct vector passing fails. There is no natural, shareable alignment baseline between two semantic spaces.

    3. Why the two existing approaches are not enough

    Two families of solutions are common for this alignment problem.

    The first is to train a dedicated mapping module. It works like a translator that explicitly learns how to project model A’s internal state into a form model B can consume. It generalizes reasonably well, but it introduces an extra component that must be trained, maintained, and kept aligned — with additional data, compute, and engineering complexity. When a model is swapped out, the translator typically has to be retrained.

    The second is to require the participating models to be structurally as similar as possible. Only when the “dialects” are close enough can internal representations be compared approximately. This sets a barrier at the entrance to the collaboration circle: models with large structural differences cannot work together, which significantly narrows the usable range.

    In short, the former trades extra training for generality, and the latter trades structural constraints for comparability. Is there a third path — one that neither trains a mapper nor requires homogeneity?

    4. A different intermediary: verify vocabulary overlap first

    The key insight here is that internal representations across models differ enormously, but their vocabularies often overlap heavily. Most mainstream models are trained on similar corpora and therefore share a large set of identical tokens and lexical items. Symbols such as “red,” “speed of light,” or “database” are recognizable common concepts for both A and B.

    Note carefully, though: “often overlapping” does not mean “always overlapping.” Before putting this into production, you must verify the actual vocabulary overlap rather than assume it:

    • Measure token coverage: sample real communication content and compute the proportion of tokens both sides can recognize.
    • Observe the OOV rate (proportion of out-of-vocabulary tokens): if the target language or specialized terminology produces a high OOV rate, the shared anchor set shrinks.
    • Check the tokenization: different models segment text differently. Where necessary, unify tokenization first or build a symbol mapping table so both sides’ tokens are aligned onto the same set of anchors.

    Only when the overlap passes the bar does “align the symbols” become viable. At that point, the strategy shifts from aligning vectors to aligning symbols:

    • Each model maintains its own “symbol → internal representation” lookup table. The same word has A’s vector in A’s space and B’s vector in B’s space.
    • When models communicate, they no longer pass high-dimensional vectors but a set of discrete symbols (words/tokens).
    • Once the receiver has the symbols, it uses its own lookup table to decode them into internal representations in its own model space.

    The symbols act as a public coordinate system or a shared index key. Neither side needs the coordinate systems themselves to align; as long as both agree on the same set of anchors, each can interpret those concepts in its own way. This sidesteps the hardest part — vector-level mapping.

    5. How the mechanism works

    State the input/output contract first; the code is easier to follow after that:

    • Input hidden: the sender’s current hidden state, shape [d], where d is this model’s hidden dimension.
    • Input vocab_emb: this model’s vocabulary embedding matrix, shape [V, d]. It must share the hidden dimension with hidden for the dot product to work. It is recommended to L2-normalize both before use so that similarity stays in a controlled range.
    • Anchor uniqueness: encode must deduplicate the top-k results to guarantee that anchor symbols do not repeat.
    • Receiver-side temporary context: during decode, the receiver builds a temporary context from the anchor symbols (for example, assembling them into a token sequence and running one forward pass), or takes the corresponding token embeddings, to obtain a representation in its own model space.
    1. import numpy as np
    2. class SymbolIndexedChannel:
    3. """Symbol-indexed channel: send symbols, not expanded high-dimensional vectors."""
    4. def __init__(self, model):
    5. self.model = model
    6. self.cache = {} # symbol -> local internal representation (lazy cache)
    7. def encode(self, hidden, vocab_emb, topk=8):
    8. """Sender: pick the top-k vocabulary symbols most relevant to the current hidden state as anchors."""
    9. # Normalize first, then compute similarity to measure "which symbols best represent the current state"
    10. h = hidden / (np.linalg.norm(hidden) + 1e-8)
    11. e = vocab_emb / (np.linalg.norm(vocab_emb, axis=1, keepdims=True) + 1e-8)
    12. scores = e @ h
    13. anchors = np.argsort(scores)[-topk:]
    14. return list(dict.fromkeys(anchors.tolist())) # deduplicate so anchors stay unique; send only discrete symbol indices
    15. def decode(self, anchors):
    16. """Receiver: build a temporary context from the symbols and take the matching local representation."""
    17. vecs = []
    18. for sym in anchors:
    19. if sym not in self.cache:
    20. # Get the vector from this model's own representation (e.g. token embedding or one conditional forward pass)
    21. self.cache[sym] = self.model.representation(sym)
    22. vecs.append(self.cache[sym])
    23. return np.mean(vecs, axis=0) # aggregate into an internal representation usable in this model's space

    The important point: encode emits a sequence of integer indices (symbols), so the communication payload is very small; decode happens entirely inside the receiver’s own representation space. The transport layer only needs to guarantee that symbols arrive correctly and unambiguously.

    Comparison checklist for the three routes

    Dimension Natural language Trained mapper Shared symbols (this approach)
    Training required No Yes, one per model pair No mapper training needed
    Structural constraint Weak Medium Weak (no isomorphism required)
    Communication cost Medium (token sequence) Low (vectors) Low (discrete symbols)
    Information loss Medium (text is lossy) Low Medium (discrete symbols are lossy)
    Pre-launch validation Prompts and parsing Mapper accuracy Vocabulary overlap, anchor count, quality evaluation

    6. Where this approach pays off, and where it breaks

    Start with the benefits, which come down to three points:

    1. No mapper training required. There is no need to train a translator for every model pair, which naturally supports plug-and-play multi-party collaboration. Note that this does not mean zero cost: vocabulary alignment, anchor selection, and quality evaluation still have to be handled.
    2. Weak structural constraints. Models do not have to be homogeneous, so alignment works across heterogeneous models and the set of models that can collaborate expands.
    3. Lightweight communication. What travels is a set of discrete symbols rather than high-dimensional continuous vectors, so bandwidth and latency costs are lower.

    Now the boundaries and failure modes, which determine when you should fall back:

    • Insufficient vocabulary overlap. If the two models’ symbol systems differ too much (radically different tokenization or language coverage), the shared anchor set shrinks and results degrade. Align at the symbol layer first, or fall back to natural language.
    • Discrete symbols are lossy. Encoding continuous, information-rich internal states with a finite symbol set is a form of compression, and how much detail survives depends on the anchor strategy and count.
    • Fallback strategy (important). When facts involve continuous numeric values, complex constraints, or ordering information (exact decimals, multi-condition logic, long sequential steps), symbol encoding tends to lose precision. In these cases fall back to natural language, or train a mapper specifically for that subtask.
    • Anchor selection tradeoff. Too few anchors may drop key semantics; too many erodes the “lightweight” advantage. Balance information content against overhead according to the task.

    7. Deployment checklist

    Before adopting this for real, work through the list item by item:

    • Data preparation: sample real communication content, measure token coverage and OOV rate, and confirm the vocabulary overlap meets the bar.
    • Anchor selection: decide the top-k value, and verify that deduplication and normalization are working.
    • Quality evaluation: define what fact-transfer accuracy means (for example, key-information hit rate or downstream task success rate) and run an A/B comparison against natural language.
    • Fallback strategy: prepare a natural language fallback or a mapper-based approach in advance for continuous numeric values, complex constraints, and ordering information.
    • Quality control: keep human review or a natural language safety net for high-risk facts, so that symbol compression cannot silently distort them.

    8. Closing

    The bottleneck in multi-model collaboration looks like a question of “how to pass information,” but it is really a question of “how to reach consensus between two different semantic spaces.” Using shared symbols as the intermediary points to a simple and effective path: instead of struggling to align high-dimensional vectors, step back to the discrete vocabulary layer that models already share, use it as the public anchor, and let each model interpret it its own way. The prerequisites are to verify vocabulary overlap, choose anchors well, evaluate results properly, and retain fallbacks for high-risk scenarios. The approach strikes a reasonable balance between generality and deployment cost, and for low-coupling, easy-to-scale multi-model systems, it is a design pattern worth adding to the toolbox.