Hardware Scheduler for Deep Learning Task Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning inference on hardware platforms faces inefficiencies due to suboptimal scheduling of operations like convolutions, activations, and pooling, leading to less than ideal performance, particularly because existing firmware schedulers rely on costly memory transactions and are not generic across different neural network types.
Innovation Solution
A hardware-based dependency algorithm that tracks a single index for each layer type, allowing for efficient scheduling without requiring memory transactions, and is implemented in a hardware engine scheduler common to all network types, ensuring maximum utilization of Deep Learning Accelerator (DLA) circuits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If firmware schedulers are used for deep learning operations, then scheduling capability is provided, but memory transactions increase causing higher latency and reduced efficiency
Solution Approach 1:
The patent replaces the firmware-based scheduling system with a hardware-based scheduling circuit. This substitution eliminates the need for memory transactions that occur in firmware systems, as the hardware scheduler directly tracks execution layer identifiers and resolves dependencies through dedicated circuitry. The hardware scheduler uses comparison circuits and state machines to determine dependency satisfaction without requiring reads/writes to memory, thereby reducing latency and improving scheduling efficiency.
2Productivity
If firmware schedulers are used for deep learning operations, then scheduling capability is provided, but device complexity increases due to memory management requirements
Solution Approach 1:
The patent extracts the scheduling function from the firmware layer and implements it as a dedicated hardware circuit. This extraction removes the complexity of memory management from the scheduling process, as the hardware scheduler maintains its own state registers and comparison circuits that operate independently of system memory. The dependency resolution logic is implemented through dedicated hardware components rather than software routines that require memory access.
3Adaptability or versatility
If generic scheduling across different neural network types is required, then versatility is improved, but scheduling precision decreases due to lack of optimization for specific network types
Solution Approach 1:
The patent implements a universal hardware scheduler that can handle different types of neural network operations (convolutions, activations, pooling, normalization) through a unified dependency resolution mechanism. The scheduler uses a generic comparison circuit that evaluates execution layer identifiers against dependency requirements without needing to know the specific operation type. This universal approach maintains scheduling accuracy by focusing on the fundamental dependency relationships rather than operation-specific details, allowing the same hardware circuit to optimally schedule various neural network types.
Data Source
AI summary
Apparatuses, systems, and techniques for scheduling deep learning tasks in hardware are described. One accelerator circuit includes multiple fixed-function circuits that each processes a different layer type of a neural network. A scheduler circuit receives state information associated with a respective layer being processed by a respective fixed-function circuit and dependency information that indicates a layer dependency condition for the respective layer. The scheduler circuit determines that the layer dependency condition is satisfied using the state information and the dependency information and enables the fixed-function circuit to process the current layer at the respective fixed-function circuit.