AI/ML Model Inference via Chunked Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI/ML model inference on mobile devices is hindered by the need for full model download and memory constraints, leading to latency and resource issues, as devices often lack sufficient memory or disk space to load the entire model for inference.
Innovation Solution
The AI/ML model is split into smaller sub-parts or chunks, allowing for parallelized download and inference, where each chunk generates intermediate results that can be used by subsequent chunks, enabling inference before the full model is downloaded and allowing for memory-efficient processing by deleting chunks after use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the full AI/ML model is downloaded before inference, then the inference accuracy is improved, but the download time and latency increase
Solution Approach 1:
The AI/ML model is divided into multiple chunks that can be downloaded and processed incrementally. The server splits the model into manageable segments, allowing the client to download and execute inference on earlier chunks while continuing to download subsequent chunks, thereby reducing total latency while maintaining accuracy through progressive model completion.
2Productivity
If the full AI/ML model is loaded in memory, then the inference performance is improved, but the memory consumption increases
Solution Approach 1:
The model is segmented into chunks that are loaded into memory incrementally as needed for inference. This allows the system to perform inference on available chunks without requiring the entire model to reside in memory simultaneously, thus maintaining inference performance while reducing peak memory consumption.
Solution Approach 2:
Chunks that have been used for inference can be discarded from memory after their results are obtained, freeing up memory resources for subsequent chunks. This dynamic allocation and release of memory segments enables efficient memory utilization while maintaining continuous inference capability.
3Quantity of substance
If the AI/ML model is split into chunks and processed incrementally, then the memory usage is reduced, but the system complexity increases
Solution Approach 1:
A chunk manager component acts as an intermediary that coordinates the download, loading, inference execution, and memory management of model chunks. This mediator handles the complexity of chunk orchestration, allowing the rest of the system to interact with a simplified interface while the intermediary manages the intricate details of incremental model processing.
Data Source
AI summary
In one implementation, the AI/ML model is first split into several unitary chunks that correspond to sub-parts of the model. Then an aggregation of unitary chunks is made by considering the download time, inference time of unitary chunks, and/or device constraints. The first split corresponds to a first chunk of AI/ML layers that, once downloaded, is useable as is, and generates intermediate results based on some sensing/perception data. As soon as a new chunk arrives, it is used to generate new results based on the intermediate data of the previous chunk. Since download and inference are parallelized, a final result can be generated earlier than with the full sequential method. In addition, as soon as the inference ends on a chunk, this chunk may be removed from the device. Several AI/ML model split methods are provided to generate model subsets/chunks for different model architectures.