Segmented ML Model Execution for Low-Memory Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mobile devices and other battery-constrained devices face challenges in executing large machine learning models due to limited volatile memory, leading to significant latency and reduced responsiveness.
Innovation Solution
The system stores machine learning models in non-volatile memory and segments them into connected layers, allowing concurrent loading and processing of segments in volatile memory to minimize latency and manage memory efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are made larger to improve accuracy, then model performance is improved, but memory requirements increase causing latency
Solution Approach 1:
The patent divides the machine learning model into multiple segments that can be loaded into volatile memory in parallel. This segmentation allows the system to maintain large models for high accuracy while reducing the memory footprint at any given time, thereby minimizing latency by loading only necessary segments into memory.
Solution Approach 2:
The system performs preliminary actions by loading model segments into volatile memory before they are needed for inference. This pre-loading of segments into memory reduces the time required to access model data during actual processing, thereby reducing latency while maintaining the ability to handle large accurate models.
2Productivity
If more volatile memory is allocated to execute large models, then model execution capability is improved, but power consumption increases
Solution Approach 1:
The patent segments the model into smaller parts that can be selectively loaded into volatile memory. This allows the system to allocate volatile memory only when needed for active model segments, reducing overall power consumption while maintaining the capability to execute large models by loading additional segments as required.
Solution Approach 2:
The system loads only the necessary portions of the model into volatile memory rather than loading the entire model. This partial loading approach reduces the amount of volatile memory actively used and consequently reduces power consumption while still providing sufficient execution capability for the specific inference task.
3Device complexity
If model segments are loaded sequentially into memory, then memory management is simplified, but processing time increases
Solution Approach 1:
The patent implements concurrent loading of model segments into volatile memory, allowing the memory loading operation to continue uninterrupted during the inference process. This continuous action ensures that model segments are always ready for processing without idle waiting time, reducing processing time while the system manages memory through coordinated segmentation.
Solution Approach 2:
The system performs preliminary loading of model segments into memory before they are needed for processing. This pre-loading allows the actual inference to start immediately without waiting for sequential model loading, thereby reducing processing time. The memory management remains manageable through structured segmentation that enables parallel loading operations.
Data Source
AI summary
A device implementing a system to execute machine learning models from memory includes at least one processor configured to receive a request to provide an input to one or more machine learning (ML) models arranged into a graph of connected layers, the one or more ML models stored in the first type of memory. The at least one processor is further configured to divide the graph of connected layers into a plurality of segments such that at least two of the plurality of segments concurrently fits within allocated space of the second type of memory. The at least one processor is further configured to cause the input to be processed through the first segment of the plurality of segments using the second type of memory while a second segment of the plurality of segments is concurrently loaded from the first type of memory into the second type of memory.