Multi-Chip LLM Inference With Ring-Shared K-V Cache

The patent divides the large language model into multiple expert units (e.g., sparse expert units and dense expert units) that can be selectively activated. Instead of computing through all parameters, the system segments the model into specialized units and routes inputs to relevant experts, reducing computational time while maintaining accuracy.

The patent implements dynamic routing mechanisms that adaptively select which expert units to activate based on input characteristics. The routing network dynamically determines the appropriate expert units for each token, allowing the system to adjust computation depth and breadth in real-time, thereby reducing overall computation time while preserving accuracy for complex inputs.

The patent applies partial action by activating only a subset of expert units for each computation rather than using all model parameters. The routing mechanism selects the minimum necessary number of expert units required to achieve accurate processing, reducing computational cost while maintaining performance through selective engagement of model components.

The patent changes the effective parameter utilization by dynamically adjusting which model parameters are activated through the routing network. Instead of using a fixed large number of parameters, the system adapts parameter selection based on input requirements, reducing computational cost while preserving accuracy through intelligent parameter routing.

The patent merges the K-V cache memory across multiple chips into a shared resource. Instead of each chip maintaining separate caches, the system combines cache memory resources and implements unified management, reducing overall memory usage and simplifying synchronization by treating the cache as a single shared entity across the chip cluster.

The patent implements universal K-V cache sharing where the same cache memory serves multiple chips simultaneously. The cache system is designed to be multi-functional, handling read/write operations for multiple chips through a unified interface, thereby reducing memory redundancy and simplifying data synchronization across the computational system.

The patent merges K-V cache memory across multiple chips into a shared pool. Instead of duplicating cache memory on each chip, the system combines memory resources and implements shared access, significantly reducing total memory usage while maintaining the computational capacity provided by multiple chips.

The patent implements selective copying of K-V cache data between chips only when necessary, rather than maintaining full copies on all chips. The ring-type data sharing method copies data incrementally and selectively, reducing overall memory usage while ensuring data availability across the chip cluster for parallel computations.