Inference System Caching via Culled Index Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning inference systems face inefficiencies when dealing with large embedding tables due to repeated indices in index vectors, leading to unnecessary repeated fetching from persistent storage, which is slow and costly.
Innovation Solution
The formation of a culled index vector with unique indices and the use of a shared cache to store retrieved weight vectors, along with a hashed values table to track encountered indices, reduces repeated requests to persistent storage by employing a Nonvolatile Memory Express (NVME) driver for efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If repeated fetching from persistent storage is performed for repeated indices in index vectors, then complete data retrieval is achieved, but input-output operations and latency increase significantly
Solution Approach 1:
The patent extracts repeated indices from the index vector to create a culled index vector that contains only unique indices. This extraction process removes the redundant fetching operations while maintaining the ability to retrieve all necessary weight vectors, thereby improving inference efficiency without sacrificing data retrieval completeness.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing unique indices in a culled index vector before the actual weight vector retrieval. It also uses a hashed values table to pre-identify and eliminate duplicate indices, so that when weight vectors are fetched, repeated requests are avoided entirely.
2Productivity
If a shared cache is implemented to store retrieved weight vectors, then repeated requests to persistent storage are reduced, but device complexity increases
Solution Approach 1:
The patent introduces a shared cache as an intermediary between the persistent storage and the inference processing. This cache stores retrieved weight vectors and serves as a mediator that reduces direct repeated requests to persistent storage. The cache is managed through a hashed values table that tracks which weight vectors have been retrieved, providing a relatively simple implementation that improves productivity without excessive complexity.
3Productivity
If a culled index vector with unique indices is formed, then repeated fetching operations are eliminated, but processing time for index processing increases
Solution Approach 1:
The patent performs the index culling and unique index identification as a preliminary action before weight vector retrieval. By pre-processing the index vector to create a culled index vector and using the hashed values table to track encountered indices, the system eliminates repeated fetching operations. The time invested in this preliminary processing is offset by the significant reduction in subsequent I/O operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A system and method for inference using an embedding table is provided. In some embodiments, the method includes forming a culled index vector including a first index, and requesting a weight vector corresponding to the first index. The first index may be a first element of a first index vector, the first index being culled within the culled index vector.