Machine Learning Model Intermediate Output Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The execution of multi-layer machine learning models on ordered sequences of inputs is computationally expensive, particularly when models require repeated updates within tight latency and resource constraints, such as in real-time applications like robot control, due to the need for extensive memory and computational resources.
Innovation Solution
The method involves storing and reusing intermediate outputs from previous model executions instead of recomputing them, allowing for efficient generation of updated outputs by shifting inputs and reusing stored outputs from layers where the ordering of inputs does not affect the outputs, thereby reducing computational costs and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-layer machine learning models are executed on ordered sequences of inputs to maintain accuracy, then model output accuracy is improved, but computational resources and time consumption increase
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing intermediate outputs from previous model executions before they are needed for the current input sequence. When processing a new ordered sequence of inputs, the system retrieves these pre-stored intermediate outputs instead of recomputing them, thereby reducing computational resources and time while maintaining model output accuracy.
2Adaptability or versatility
If model memory is increased to handle longer input sequences, then model capability is improved, but memory consumption increases
Solution Approach 1:
The patent extracts and stores intermediate outputs from specific layers of the machine learning model separately from the main model memory. By taking out these intermediate results and storing them in a dedicated cache or memory structure, the system enables efficient retrieval for subsequent processing without increasing the overall memory footprint of the model itself, thus maintaining model capability while controlling memory consumption.
3Measurement precision
If intermediate outputs are recomputed for each input sequence, then computational accuracy is maintained, but processing time increases
Solution Approach 1:
The patent creates copies of intermediate outputs from previous model executions and stores them for reuse. Instead of recomputing these intermediate values for each new input sequence, the system retrieves and uses the copied intermediate outputs, which maintains computational accuracy (since the intermediate computations are deterministic) while dramatically reducing processing time for ordered sequences of inputs.
4Duration of action of moving object
If the number of layers and units is increased to extend model memory, then temporal context is improved, but device complexity increases
Solution Approach 1:
The patent extends the model's temporal context capability by adding a temporal dimension through caching intermediate outputs across different time steps, rather than increasing the spatial dimensions (number of layers and units). This allows the model to maintain longer temporal context by retrieving relevant intermediate outputs from previous time steps without making the model structure more complex.
Data Source
AI summary
Improved methods for executing multi-layer machine learning model architectures in the context of ordered inputs sequences that experience progressive updates are provided that exhibit increased decreased inference compute cost and/or decreased inference time latency. These improved models include storing some or all of the intermediate outputs of the model's units for later re-use, e.g., once one or more novel inputs of an input sequence have been obtained. Storing such intermediate outputs allows the computational effort used to generate them (e.g., by applying the relevant model input(s) to the relevant unit(s) and/or layer(s) of the model) to be avoided in subsequent execution of the model. Instead, only those model units whose outputs would differ from one model execution to the next are re-computed in order to generate an updated model output, thereby significantly reducing the computational cost and/or time to execute the model in light of the updated input(s).