Neural Processing Device Task Management With Reduced CPU Intervention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face challenges in efficiently managing tasks due to limitations in computing performance and task distribution, leading to increased latency and overhead from synchronization signals and CPU intervention.
Innovation Solution
A neural processing device with a neural core, core global, and task manager that efficiently manages tasks through task descriptors, completion signals, and runtime handles to minimize CPU intervention and optimize task management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural processing units use multiple computation devices operating in parallel, then computation efficiency is enhanced, but task management complexity increases
Solution Approach 1:
The system segments task management into distinct functional units: task passage for task generation and distribution, done passage for completion handling, and runtime handle for execution control. This segmentation allows parallel computation devices to operate independently while maintaining organized task management through specialized sub-components.
Solution Approach 2:
The patent introduces intermediate data structures (task descriptors, completion reports) and intermediate components (runtime handle, passage interfaces) that mediate between the command processor and parallel computation devices. These intermediaries simplify task management by handling coordination and synchronization automatically, reducing the complexity burden of parallel operation.
2Measurement precision
If task descriptors are checked in and checked out for each task, then task tracking accuracy is improved, but processing overhead increases
Solution Approach 1:
The system performs preliminary actions by pre-allocating task descriptors and setting up task passages before actual computation begins. Task descriptors are prepared and checked in advance, allowing the parallel computation devices to immediately begin processing without repeated setup overhead during execution.
Solution Approach 2:
The patent uses task descriptors as copyable data structures that can be replicated and distributed to multiple computation devices. Instead of managing complex pointers or references to the same task object, the system creates and manages independent copies of task descriptors, simplifying tracking while reducing synchronization overhead.
3Stability of the object's composition
If the runtime handle controls task descriptors to wait in the task queue, then task execution order is maintained, but task completion latency increases
Solution Approach 1:
The runtime handle dynamically adjusts task descriptor states based on execution conditions. Task descriptors can transition between waiting, executing, and completed states, allowing the system to maintain order when necessary while enabling parallel execution when tasks are independent. The wait control is dynamic rather than static, adapting to actual task dependencies.
Solution Approach 2:
The patent applies different quality controls to different task descriptors based on their specific requirements. Some task descriptors are marked to wait in the queue for strict ordering, while others can proceed immediately if they have no dependencies. This local differentiation allows the system to maintain execution order only where necessary, reducing overall latency.
Data Source
AI summary
A neural processing device and a method for managing tasks thereof are provided. The neural processing device includes a neural core configured to perform a task and generate a completion signal for completion of the task, a core global configured to transfer task information for the task to the neural core and receive the completion signal of the task from the neural core, and a task manager configured to generate and transmit the task information to the core global, receive the completion signal from the core global, generate a completion report, and transmit the completion report.