Neural Network Layer Scheduling for Memory-Constrained Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Wearable devices with limited runtime memory struggle to execute multiple deep learning models due to their large size and complexity, leading to feasibility issues with loading and executing these models locally, which worsens as models become more complex.
Innovation Solution
Implementing a scheduler that classifies neural network layers into computation-heavy and memory-heavy sets, allowing for parallel execution and loading of these layers into a smaller, faster memory, reducing the need for loading full models and minimizing inference time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple deep learning models are loaded into runtime memory simultaneously, then more models can be executed for comprehensive vision tasks, but the limited runtime memory (less than 1 GB) cannot accommodate multiple large models (each nearly 500 MB)
Solution Approach 1:
The patent divides each deep learning model into multiple layers and segments the execution process into loading individual layers rather than full models. The scheduler loads only the necessary layers into runtime memory at a time, enabling multiple models to coexist without requiring all models to be fully loaded simultaneously, thus resolving the memory capacity constraint.
Solution Approach 2:
The patent introduces a temporal dimension to model loading by using asynchronous loading mechanisms. Instead of loading all models simultaneously in a single dimension, the system loads models layer-by-layer over time, allowing multiple models to be managed in memory through time-multiplexed access patterns.
2Reliability
If full deep learning models are loaded into runtime memory, then complete model functionality is available, but the models become too large to fit into limited runtime memory
Solution Approach 1:
The patent segments models into layers and implements a scheduler that loads only the necessary layers into runtime memory. This allows the system to maintain complete model functionality by loading layers on-demand while keeping the memory footprint manageable, thus resolving the contradiction between model completeness and memory constraints.
Solution Approach 2:
The system loads only the partial layers needed for current inference tasks rather than full models. The scheduler determines which layers are required based on the active models and loads only those, avoiding the excessive action of loading complete models when only portions are necessary.
3Quantity of substance
If models are loaded sequentially one by one, then memory resources are optimized for single model execution, but the paging of models in and out of memory consumes significant time and increases overall inference time
Solution Approach 1:
The patent implements asynchronous loading where layers are loaded into memory in advance before they are needed for inference. The scheduler pre-loads necessary layers for upcoming model executions, eliminating the time-consuming sequential paging operation and allowing continuous inference without interruption.
Solution Approach 2:
The system maintains continuous model execution by overlapping layer loading operations with inference operations. While one model is executing, the scheduler asynchronously loads layers for the next model, ensuring continuous useful action without idle paging time, thus reducing overall inference time while optimizing memory utilization.
4Measurement precision
If deep learning models become more complex with more layers, then model accuracy and capability improve, but the models become more memory intensive and harder to execute on resource-constrained devices
Solution Approach 1:
The patent segments complex models with many layers into manageable units that can be loaded and executed individually. The scheduler manages these segmented layers, allowing the system to handle more complex and accurate models by processing them in smaller, memory-efficient chunks rather than requiring the entire complex model to reside in memory at once.
Data Source
Figure 1~3
AI summary
Embodiments relate to a method for analyzing sensor data, executed by an analyzing device (1) comprising a processor (2), a first memory (3) and a second memory (4), wherein the processor (2) is capable of executing multiple threads in parallel in function of instructions and/or data stored in the first memory (3), wherein the second memory (4) stores a plurality of neural networks, a neural network specifying successive layers, the method comprising determining respective analysis results by processing the sensor data with the neural networks, wherein processing the sensor data with the neural networks comprises, for the respective neural networks: - classifying (S1-S4) the layers of a neural network into a first set of layers and a second set of layers, wherein the second set of layers follows the first set of layers, - executing (T2, T3) in parallel, by the processor (2): - a first thread which comprises loading said first set of layers in the first memory (3) and executing said first set of layers, and - a second thread which comprises loading said second set of layers in the first memory (3), - after finishing execution of the first thread, executing (T5) the second set of layers.