Researchers propose Mixture-of-Translators (MoT) to reuse key-value caches across different LLM architectures, addressing the inefficiency of redundant prefills in multi-model systems. Unlike single-projection methods, MoT employs multiple translator modules to capture diverse mappings between source and target cache spaces. This approach aims to improve scalability for shared contexts and multi-agent dialogues by eliminating the need for each model to independently process identical context data.
- Enables KV cache reuse across heterogeneous LLMs, reducing redundant computation.
- Uses multiple translators instead of a single projection for better mapping accuracy.
- Improves scalability for multi-agent systems and shared context retrieval.
- Mitigates storage overhead by avoiding duplicate cache storage per model.