ML Compute Graph Compilation for Edge Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Edge devices have limited processing hardware, memory hardware, battery power, and network interface capabilities, making it challenging to implement machine learning models, particularly due to memory constraints during inference processes.
Innovation Solution
A system and method for obtaining a compute graph associated with a machine learning model, determining a memory allocation scheme, and compiling the model to minimize memory requirements for execution on edge devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are deployed on edge devices, then applications can perform local inference with lower latency and improved privacy, but the limited memory hardware of edge devices cannot accommodate the large memory requirements of machine learning models during inference
Solution Approach 1:
The patent segments the machine learning model execution into distinct phases (e.g., weight loading, inference computation, result output) and allocates memory dynamically to each phase. During inference, only the necessary weights and input data are loaded into memory, while other data is stored externally. This segmentation allows the model to run on edge devices with limited memory by breaking down the memory-intensive operations into manageable chunks that fit within device constraints.
2Measurement precision
If machine learning models are trained with high accuracy requirements, then the model performance improves, but the model size and memory requirements increase, making deployment on resource-constrained edge devices difficult
Solution Approach 1:
The patent applies local quality by differentiating between weights that are frequently accessed during inference and those that are not. Frequently accessed weights are kept in device memory, while less frequently accessed weights are stored externally. This selective memory allocation maintains model accuracy by ensuring that critical weights are available in fast memory, while reducing overall memory usage by offloading less critical data to external storage.
3Speed
If the machine learning model is optimized for high performance computation, then inference speed improves, but the memory requirements increase due to caching and buffering requirements
Solution Approach 1:
The patent implements dynamic memory allocation where the buffer size and memory allocation are adjusted based on the current inference workload and available device resources. During peak inference operations, larger buffers are allocated to maintain high throughput, while during lower workload periods, memory is freed up. This dynamic approach allows the system to achieve high inference speeds when needed while adapting to the limited memory constraints of edge devices.
Data Source
AI summary
In some aspects, a processor of one or more computing machines obtains a compute graph associated with a machine learning model. The processor determines, based on the compute graph, a memory allocation scheme associated with a configuration for executing the machine learning model on an edge device. The processor compiles the machine learning model to generate a compiled machine learning model. The processor may compile the machine learning model based on the memory allocation scheme.