Neural Network Inference Parameter Caching for Faster Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The delay in querying sparse parameters from a distributed database in a computing cluster for neural network models is high, making it difficult to meet the requirements for efficient inference task execution.
Innovation Solution
Implement a local memory with high-speed read/write capabilities in the inference server to query parameters first, followed by a memory pool for parameters not found in the local memory, utilizing a shared memory managed by a hash table for efficient and concurrent access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a distributed database is used to store sparse parameters in a computing cluster, then parameter storage capacity is improved, but parameter query delay increases
Solution Approach 1:
The patent segments the parameter storage system into multiple levels: a local cache memory layer for frequently accessed parameters, a shared memory layer for general parameter storage, and a distributed database layer for comprehensive parameter storage. This segmentation allows the system to maintain large parameter storage capacity while reducing query delay by serving frequently accessed parameters from faster local storage layers.
Solution Approach 2:
The patent implements a nested storage architecture where the local cache memory is nested within the inference server, the shared memory is nested within the computing cluster, and the distributed database is nested at the cluster level. This nested structure enables hierarchical access to parameters, allowing the system to achieve both high storage capacity and low query delay by accessing parameters from appropriate nested layers.
2Adaptability or versatility
If a query service is provided by a distributed database, then parameter accessibility is improved, but inference task execution efficiency deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-loading frequently accessed sparse parameters into the local cache memory before they are needed for inference tasks. This is achieved through a parameter prediction mechanism that identifies likely-to-be-accessed parameters and loads them in advance, thereby improving inference task execution efficiency while maintaining parameter accessibility through the multi-level storage system.
Solution Approach 2:
The patent introduces an intermediary parameter caching layer between the distributed database and the inference cards. This intermediary layer acts as a buffer that reduces the frequency of direct database queries, thereby improving inference execution efficiency while maintaining the accessibility benefits of the distributed database through the hierarchical architecture.
3Quantity of substance
If only some model parameters are stored in the distributed database, then storage cost is reduced, but parameter retrieval speed deteriorates
Solution Approach 1:
The patent applies local quality by storing different types of parameters in different locations based on their access characteristics. Frequently accessed parameters are stored in the local cache memory with high retrieval speed, while less frequently accessed parameters are stored in the distributed database with lower cost. This differentiated storage strategy optimizes both storage cost and parameter retrieval speed for critical parameters.
Solution Approach 2:
The patent implements partial action by selectively storing only the most important and frequently accessed parameters in the local cache memory and shared memory, rather than duplicating all parameters at every level. This partial storage approach reduces storage costs while maintaining fast retrieval speeds for the critical parameters that matter most for inference task execution.
Data Source
AI summary
A method is applied to a computing cluster. The computing cluster includes a plurality of inference servers and a memory pool. Each inference server includes at least one inference card and a local memory. The method includes: a first inference card of a first inference server in the computing cluster receives an inference task. If a parameter for executing the inference task is not hit in the first inference card, the first inference card obtains the parameter from a local memory of the first server. If the parameter is not hit in the local memory of the first server, the first inference card obtains the parameter from the memory pool. The first inference card can execute the inference task based on all obtained parameters.