Task Queue Scheduling for Heterogeneous ML Compute Circuits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current approaches for scheduling tasks on heterogeneous compute circuits in deep learning applications are inefficient, particularly in scaling and maintaining complex neural network graphs, leading to performance bottlenecks and errors due to the complexity of defining and deploying application graphs.
Innovation Solution
A method and system that instantiate kernel objects based on input definitions, generate task graphs with nodes and edges representing tasks and data dependencies, and utilize task queues to assign and manage tasks across multiple compute circuits, enabling efficient scheduling and processing of tasks on heterogeneous compute circuits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tasks are scheduled on heterogeneous compute circuits using traditional methods, then processing can be performed, but processing throughput is limited due to inefficiency in scheduling and scaling
Solution Approach 1:
The patent introduces a task queue as an intermediary data structure between the application graph and heterogeneous compute circuits. The task queue abstracts the complexity of scheduling by providing a standardized interface for enqueueing tasks and retrieving them for execution on appropriate compute circuits, thereby improving throughput without requiring direct management of complex scheduling logic in the application graph
Solution Approach 2:
The patent segments the application graph into discrete tasks that can be independently enqueued and executed on different heterogeneous compute circuits. By breaking down the neural network processing into separable tasks with defined data dependencies, the system can scale across multiple compute circuits while maintaining manageable complexity through the task queue abstraction
2Quantity of substance
If the system scales to handle complex neural network graphs, then more tasks can be processed, but performance bottlenecks and errors increase due to scheduling inefficiency
Solution Approach 1:
The task queue serves as a mediator that manages the flow of tasks between the application graph and compute circuits, preventing performance bottlenecks by decoupling task generation from task execution. This abstraction layer ensures reliable scheduling by handling task assignment to appropriate compute circuits without requiring the application graph to directly manage scheduling complexity
Solution Approach 2:
The system implements feedback mechanisms through the task queue management, where completed tasks are retrieved and their results fed back into the system for further processing or output. This feedback loop enables the system to handle complex neural network graphs reliably by maintaining proper data flow and dependency management across multiple compute circuits
Data Source
AI summary
Scheduling work of a machine learning application includes instantiating kernel objects by a computer processor in response to input of kernel definitions. Each kernel object is of a kernel type indicating a compute circuit. The computer processor generates a graph in a memory. Each node represents a task and specifies an assignment of the task to one or more of the kernel objects, and each edge represents a data dependency. Task queues are created in the memory and assigned to queue tasks represented by the nodes. Kernel objects are assigned to the task queues, and the tasks are enqueued by threads executing the kernel objects, based on assignments of the kernel objects to the task queues and assignments of the tasks to the kernel objects. Tasks are dequeued by the threads, and the compute circuits are activated to initiate processing of the dequeued tasks.