Neural Signal Processor Multi-Graph Scheduling for Thread Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for executing multiple graphs on processing devices result in significant underutilization of available resources, leading to inefficient and prolonged execution times due to sequential execution of graphs, where one graph is completed before another begins.
Innovation Solution
Simultaneous execution of multiple graphs is achieved by sharing processing threads across graphs and varying graph priorities, allowing each thread to execute available operations from different graphs if no operations are ready on the prioritized graph, thereby optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential execution of graphs is used, then graph execution order is simple to manage, but processing resource utilization is poor and execution time is prolonged
Solution Approach 1:
The patent implements dynamic graph prioritization where thread priorities are not fixed but can be adjusted based on current execution state. The system dynamically switches between sequential and parallel execution modes, and dynamically adjusts thread priorities during execution to optimize resource utilization while managing complexity adaptively
Solution Approach 2:
The patent segments the execution of multiple graphs by dividing them into phases and operations that can be executed in parallel. By breaking down graphs into smaller executable units and assigning different threads to different graphs with varying priorities, the system achieves parallel execution while maintaining manageable scheduling through structured segmentation
2Productivity
If parallel execution of multiple graphs is implemented, then resource utilization improves, but scheduling complexity increases
Solution Approach 1:
The system employs dynamic priority adjustment where thread priorities are adapted during execution based on graph completion status and resource availability. This dynamic approach allows parallel execution with improved resource utilization while avoiding the need for complex static scheduling algorithms
Solution Approach 2:
The scheduling system uses self-service mechanisms where threads automatically adjust their behavior based on the execution state of other threads and graphs. The system monitors resource utilization and automatically rebalances thread priorities without external intervention, reducing scheduling complexity while maintaining high resource utilization
3Loss of time
If thread priorities are varied across graphs, then execution time disparity is reduced, but control complexity increases
Solution Approach 1:
The patent implements dynamic priority adjustment where thread priorities are changed during execution based on the completion status of graphs. This allows the system to reduce execution time disparity by giving higher priority to graphs that are falling behind, while maintaining simple control through rule-based dynamic adjustment rather than complex predictive algorithms
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for multi-graph execution in a processing system. Embodiments include receiving, by the processing system, a first graph representing operations related to a first machine learning model and a second graph representing operations related to a second machine learning model. Embodiments include prioritizing, by a first shared thread of the processing system, execution-ready operations from the first graph over execution-ready operations from the second graph. Embodiments include prioritizing, by a second shared thread of the processing system, execution-ready operations from the second graph over execution-ready operations from the first graph. Embodiments include executing, by the first shared thread and the second shared thread, respective operations related to the first machine learning model and the second machine learning model based on the prioritizing by the first shared thread and the prioritizing by the second shared thread.