Inference Cache Scoring for Variable Runtime Model Outputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing caching strategies for deterministic models, such as LRU and LFU, are suboptimal when inference execution time varies widely among samples, as they focus on easy recalculations rather than computational power requirements, and require prior knowledge of data distribution to optimize cache behavior.
Innovation Solution
A caching strategy based on execution time, frequency, and order of input samples, using a Bayesian optimization algorithm to dynamically adjust scaling parameters without prior knowledge of the dataset distribution, optimizing cache performance by calculating a weighted score for each element.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If LRU or LFU caching strategy is used, then cache implementation is simple, but cache performance is suboptimal when execution time varies widely among samples
Solution Approach 1:
The patent changes the caching parameter from simple recency (LRU) or frequency (LFU) to a composite parameter that incorporates execution time. The cache eviction decision is based on a score that includes execution time as a key factor, transforming the caching strategy from static rules to dynamic, performance-aware decisions.
Solution Approach 2:
The patent introduces dynamic adaptation by learning the data distribution and adjusting caching parameters accordingly. The system transitions from fixed LRU/LFU policies to a dynamic strategy that adapts to the specific characteristics of the workload, including varying execution times and data patterns.
2Productivity
If execution time based caching is used, then cache performance improves for some workloads, but performance degrades for workloads where frequency and order matter
Solution Approach 1:
The patent merges multiple caching criteria into a unified scoring mechanism. The eviction score combines execution time, frequency, and recency components, allowing the system to simultaneously consider multiple factors that matter for different workload types. This composite approach enables the cache to adapt to various workload characteristics.
Solution Approach 2:
The patent implements feedback through the learning component that observes data distribution patterns and adjusts caching parameters accordingly. The system continuously learns from actual cache performance and data characteristics, refining its eviction decisions to optimize for the specific workload at hand.
3Productivity
If data distribution learning is performed to optimize caching parameters, then cache behavior is optimized, but additional time is required to learn the distribution
Solution Approach 1:
The patent performs preliminary learning of data distribution characteristics during an initial phase or warm-up period. By learning the distribution upfront, the system establishes optimized caching parameters before full operational load, trading initial setup time for long-term performance gains.
Solution Approach 2:
The patent implements partial learning by focusing on the most critical aspects of data distribution rather than complete characterization. The learning mechanism captures essential patterns needed for effective caching without requiring exhaustive analysis of all data characteristics.
Data Source
AI summary
A computer-implemented method includes receiving an input for a model from a data stream, computing an output from the model, and storing the input and the output as an element of a cache. The method also includes using an algorithm to determine a set of parameters associated with the cache; the algorithm optimizes a function including a time taken by the model to generate outputs from a set of inputs sampled from the data stream. The method further includes calculating a caching score associated with each cache element, based on the set of parameters and the time taken by the model to generate the output, a usage of the element expressed as a number of iterations over which the element has been retained in the cache, and a frequency of usage of the element. The method also includes subsequently removing from the cache the element having the lowest caching score.