AI Accelerator Scheduler Parallel Inference Weight Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI hardware accelerators face challenges in efficiently processing inference tasks in parallel due to limitations in resource allocation and scheduling algorithms, leading to suboptimal performance and resource utilization.
Innovation Solution
The proposed method involves a scheduler that determines whether any group shares weights with a first group, identifies idle processing element (PE) arrays, and selects a second group with overlapping memory and computation times to optimize resource allocation and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a scheduler processes inference tasks sequentially in existing AI hardware accelerators, then resource allocation is simplified, but processing time and resource utilization deteriorate
Solution Approach 1:
The patent segments inference tasks into multiple groups that can be processed in parallel. The scheduler divides the workload into independent task groups, each assignable to different PE arrays, enabling concurrent execution and improving throughput without exponentially increasing scheduler complexity
Solution Approach 2:
The patent performs preliminary actions by pre-calculating task group dependencies, memory requirements, and execution schedules before actual processing begins. This advance planning enables efficient parallel execution while keeping the runtime scheduler relatively simple
2Productivity
If multiple groups are processed in parallel using idle PE arrays, then resource utilization improves, but scheduling complexity and memory management difficulty increase
Solution Approach 1:
The patent implements dynamic scheduling where the scheduler adapts to available PE arrays and task dependencies in real-time. The system dynamically assigns tasks to idle PE arrays based on current resource availability and task readiness, optimizing resource utilization without requiring overly complex static scheduling
Solution Approach 2:
The patent uses weight sharing between task groups, where multiple groups can reference the same weight parameters stored in memory. This copying approach allows parallel processing of multiple groups without proportionally increasing memory requirements, as shared weights are reused across groups
3Quantity of substance
If weight sharing between groups is implemented, then memory usage is reduced, but determining group compatibility and scheduling difficulty increase
Solution Approach 1:
The patent implements feedback mechanisms where the scheduler continuously monitors which weights are shared across task groups and adjusts scheduling decisions accordingly. This feedback loop helps manage the complexity of determining group compatibility by providing real-time information about weight sharing patterns and memory usage
Data Source
AI summary
Disclosed is an accelerator and a method of operating the accelerator including determining whether any group shares weights of a first group from among groups, determining a presence of an idle processing element (PE) array, in response to no group sharing the weights of the first group, and selecting a second group having a memory time overlapping a computation time of the first group from among the groups, in response to the idle PE array being present.