ML Inference Segmentation for Jitter Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently processing inference operations of multiple machine learning models within a limited time window, leading to issues like jitter in data communication, idle time for hardware accelerators, and unsatisfactory quality of service and user experience.
Innovation Solution
A method that determines the priority of multiple machine learning models, reserves time for high-priority tasks, and partitions lower-priority models into sub-models to fit within the remaining time window, ensuring efficient processing and reduced jitter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system processes inference operations of multiple machine learning models within a limited time window, then the productivity is improved, but the jitter in data communication increases
Solution Approach 1:
The patent segments the time window into multiple priority levels (first priority time period, second priority time period, third priority time period) and divides machine learning models into different groups based on their priority. This segmentation allows the system to process high-priority models first within their designated time periods, ensuring timely processing and reducing jitter, while still fitting multiple models within the overall time window to maintain high productivity.
2Manufacturing precision
If the system processes larger and deeper neural networks to improve performance, then the manufacturing precision is improved, but the device complexity increases
Solution Approach 1:
The patent introduces dynamic priority assignment for machine learning models based on their characteristics and requirements. The system can dynamically adjust which models are processed in which priority time periods, allowing flexible accommodation of complex models without rigidly increasing system complexity. This dynamic approach enables the system to handle larger and deeper neural networks while maintaining manageable operational complexity.
3Reliability
If the system reserves time for high-priority inference operations, then the reliability is improved, but the productivity decreases
Solution Approach 1:
The patent implements periodic action by organizing inference operations into recurring time windows with structured priority levels. Each time window contains designated periods for different priority models, creating a rhythmic processing pattern. This periodic structure ensures that high-priority models receive guaranteed processing time (improving reliability) while the systematic arrangement allows multiple models to be processed efficiently within each cycle (maintaining productivity).
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing inference operations of machine learning models, are described in this document. In one aspect, the method includes receiving data representing a first machine learning model that includes inference operations. An estimated duration for the system to perform the inference operations is obtained. A priority time period reserved for performing priority inference operations of a priority machine learning model during each occurrence of a recurring time window is obtained. A remaining time period of each occurrence of the recurring time window that remains after reserving the priority time period is determined. A determination is made that the estimated duration is greater than the remaining time period. In response, the first machine learning model is partitioned into a group of sub-models. The hardware processing unit(s) perform inference operations of a sub-model during the remaining time period.