EFFICIENT MODEL AGENT COMMUNICATION WITH CACHE SHARING

Fecha de publicación: --
Fuente: WIPO "tomato"
Example solutions for context sharing between GAI models includes: identifying a pair of GAI models including a first model and a second model; generating a first key-value (KV) cache of the first model; populating a second KV cache of the second model with one or more layers of the first KV cache; recomputing one or more other layers of the second KV cache; computing an output performance score for the second model based on a baseline metric; and transmitting a reuse pattern to a host device for use in context sharing between an instance of first model and an instance of second model, the reuse pattern identifying the plurality of first KV cache layers to be reused, thereby causing the host device to generate a local KV cache for the instance of second model by reusing the plurality of first KV cache layers from a KV cache of first model.