Pass Symbols, Not Vectors: How Heterogeneous LLMs Can Share Facts
Multi-model systems often stall on communication: natural language is slow and lossy, while raw hidden states do not transfer across architectures. This article explains why vector exchange fails, reviews two existing approaches and their trade-offs, and describes a third path—sharing public symbols while keeping private representations local.
The communication bottleneck in multi-model systems
Orchestrating several language models has become a common way to build complex AI systems: one model retrieves, another reasons, another writes code. Combined, they can outperform any single model on large tasks.
Collaboration depends on communication. Most systems still talk in natural language—token generation, decoding to text, then re-encoding. That loop is readable and debuggable, but as tasks grow and call rates rise it is low-density, high-latency, and prone to losing detail in paraphrase.
A more aggressive idea is to skip text and exchange internal representations (hidden states or activations) directly. In theory that is faster and closer to “thought-to-thought” transfer. Making it work means clearing a structural obstacle first.
The core problem: incompatible internal spaces
Models differ in architecture, dimension, and learned geometry. The same concept occupies different coordinates in each model. Model A’s vector for “red” is meaningless in B’s space—even the dimensions may not match. Passing the numbers unchanged is like playing a Chinese audio waveform to someone who only reads English: the physical signal arrives, the semantics do not.
Meaning is not in the numbers; it is in each model’s privately trained coordinate system. Cross-model vector communication is therefore not “how to send a tensor,” but how to bridge two private representation spaces.
Two existing routes and their costs
Research has mainly followed two paths.
The first trains a dedicated translator—for example cache-to-cache mappers that convert source hidden states into a form the target can use. Cost: extra training, paired data, and typically one mapper per model pair. Change the combination and you retrain.
The second assumes participants are similar enough in architecture or representation that internal states can be aligned without a mapper. That avoids training but sharply limits which models can collaborate—often the opposite of what practitioners want when pairing dissimilar systems.
In short: training cost versus generality, or architectural similarity versus no training. Is there a third option that needs neither a trained translator nor similar structure?
A third path: pass symbols, not vectors
One line of work can be summarized as: do not pass vectors; pass words.
The inspiration is that a given word (e.g. “red”) corresponds to a relatively stable internal representation that supports many tasks, not a single isolated one. Words act as semantic anchors—shared, stable units inside a model.
The protocol is then: each model keeps a local symbol-to-vector table for a shared set of public symbols (words or identifiers). The sender never needs the receiver’s internals; it only sends a symbol such as “red.” The receiver looks up its own vector for that symbol and injects it into its hidden state.
Schematic flow:
# Sender: map the fact to a public symbolsymbol = sender.encode_to_symbol(fact) # e.g. the word "red"sender.send(symbol) # only the symbol is transmitted# Receiver: translate locallysymbol = receiver.recv()local_vector = receiver.lookup(symbol) # local "symbol → private vector" tablereceiver.inject(local_vector) # write into own hidden state
Vectors never leave their models. Only symbols cross the wire. Alignment happens at the abstract symbol layer, not at comparable physical coordinates.
Why symbols work better
This design avoids both earlier pitfalls.
First, no translator network: encoding is on the sender; lookup uses the model’s existing word–vector mapping.
Second, architectures need not match. Different depths and dimensions are fine if they share a symbol set. Alignment is symbolic, not geometric.
Third, adding a third model means joining the same public symbol system—no new pairwise mapper. Collaboration becomes closer to plug-and-play.
Fourth, discrete, shared symbols are a minimal common protocol for heterogeneous systems, analogous to a shared character encoding between machines that do not share memory layouts.
Engineering concerns
Symbol-space alignment is foundational: who maintains the shared vocabulary, and how to keep interpretations consistent. Coarse symbols crush meaning; fine ones raise alignment cost.
Ambiguity and context matter. Polysemy means an isolated token can drop context. Systems may add context tags or fall back to natural language when symbols are not enough—a hybrid channel.
Natural language remains useful for readability, debugging, and complex intent. Symbols suit high-frequency, structured state transfer; text is the fallback for human review.
Observability changes: logs must track encode, transmit, lookup, and inject so misunderstandings can be traced.
Conclusion
Heterogeneous LLM collaboration is blocked by private, incompatible vectors. Training expensive translators or forcing architectural similarity are limited. Sharing symbols while keeping vectors local is a cleaner third path: each model guards its representation space and receives facts by looking up a public protocol.
The design principle is general: when internal states cannot be aligned, agree on a shared abstract layer rather than copying private coordinates. That idea applies to any loosely coupled multi-component system.
