Inference Model Preemption at Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of deactivating and reactivating lower-priority inference models in favor of higher-priority models in embedded systems is resource-intensive, consuming memory and processing cycles due to the need to save and reload context data, especially when preempted at layers with large context sizes.
Innovation Solution
Implementing a method to preempt and resume inference models at optimal boundaries, where minimal data needs to be stored off-chip, by determining preemptable boundaries based on layer statistics and breathing times to minimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a lower-priority inference model is deactivated in favor of a higher-priority model, then the higher-priority model can execute, but the deactivated model's context must be saved to memory, consuming valuable memory and processing cycles
Solution Approach 1:
The system performs preliminary identification of preemptable boundaries within the inference model execution flow. By pre-determining where preemptions can occur without disrupting critical operations, the system prepares the execution context in advance, allowing for efficient saving and restoration of state when preemptions are necessary.
Solution Approach 2:
The system dynamically changes execution parameters by identifying specific boundary points in the inference model where preemption is safe. These boundaries represent optimal points where the model's internal state can be saved with minimal memory overhead and where resumption can occur without loss of computational integrity.
2Adaptability or versatility
If context is saved to memory when an inference model is preempted, then the model can be restarted, but this reduces the amount of memory available to other processes
Solution Approach 1:
The system extracts only the essential context information needed for model resumption at preemptable boundaries. By identifying minimal state requirements at these boundaries, the system saves only what is necessary, leaving maximum memory available for other processes while maintaining the ability to resume execution.
3Reliability
If preemption occurs at a layer with large context size, then the higher-priority model can execute, but the saving and restoration of context consumes valuable processing cycles
Solution Approach 1:
The system performs preliminary identification of preemptable boundaries that minimize context size. By analyzing the inference model structure in advance, the system identifies points where the computational state is smallest, allowing for rapid saving and restoration operations that minimize processing cycle overhead.
Solution Approach 2:
The system applies different preemption strategies at different locations within the inference model. By identifying local boundaries with favorable characteristics (smaller context sizes, natural synchronization points), the system optimizes the preemption process at each specific location rather than using a uniform approach.
Data Source
AI summary
Disclosed herein are systems and methods for inference model scheduling of a multi priority inference model system. A processor determines an interrupt flag has been set indicative of a request to interrupt execution of a first inference model in favor of a second inference model. In response to determining that the interrupt flag has been set, the processor determines a state of the execution of the first inference model based on one or more factors. In response to determining the state of the execution is at a preemptable boundary, the processor deactivates the first inference model and activates the second inference model.