Deterministic Model Cache Scoring for Execution-Time-Aware Eviction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing caching strategies like LRU and LFU are suboptimal for datasets with repetitive samples due to their focus on easy recalculations rather than computational power requirements, execution time, frequency, and input order, leading to inefficient use of computational resources.
Innovation Solution
A caching strategy based on execution time, frequency, and input order, using a weighted score (sc=αst+βsr+γsf) to prioritize cache entries, where st, sr, and sf represent normalized scores for execution time, order, and frequency, respectively, with α, β, and γ as scaling weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If LRU or LFU caching strategy is used, then cache implementation is simple, but cache performance is suboptimal for datasets with varying execution times
Solution Approach 1:
The patent changes the parameters used for cache eviction from simple LRU (time-based) or LFU (frequency-based) to a composite parameter that incorporates execution time, frequency, and recency. This allows the cache to prioritize entries based on actual computational value rather than simple metrics, resolving the contradiction between implementation simplicity and performance optimization.
Solution Approach 2:
The patent creates a composite caching strategy that combines multiple factors (execution time, frequency, recency) into a unified eviction policy. This composite approach leverages the strengths of individual strategies while mitigating their weaknesses, achieving better overall cache performance without excessive complexity.
2Ease of operation
If cache focuses on least recently used or least frequently used elements, then eviction policy is simple, but computational resources are wasted on easy recalculations
Solution Approach 1:
The patent introduces execution time as a critical parameter in the eviction decision process. By weighting entries with longer execution times more heavily, the cache prioritizes retaining computationally expensive results, thereby reducing energy waste on recalculations while maintaining a manageable eviction policy.
Solution Approach 2:
The cache system incorporates feedback from actual model execution performance. By monitoring which cache entries result in significant computational savings, the system dynamically adjusts eviction priorities to maximize resource efficiency while keeping the policy operational simple.
3Productivity
If cache prioritizes elements requiring greater computational power, then computational efficiency improves, but cache management complexity increases
Solution Approach 1:
The patent modifies the cache management parameters to include weighted combinations of execution time, frequency, and recency. This allows the system to prioritize computationally valuable entries while using a structured scoring mechanism that prevents unmanageable complexity in the eviction logic.
Data Source
AI summary
A computer-implemented method includes receiving an input for a deterministic model, and determining whether the input and an output from the model corresponding to the input are stored in a cache. The method further includes, in accordance with the input and the output not being stored in the cache, computing the output from the model based on the input, storing the input and the output as a cache element, and storing raw scores respectively indicating a usage frequency and order of usage of the element, and a computation time for computing the output. The method further includes calculating a caching score for the element; the caching score includes a sum of a normalized scores each corresponding to a product of a raw score and a scaling factor.