Neural Task Queue Switching for Priority-Based NPU Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning systems relying solely on central processing units (CPUs) for neural network operations consume significant bandwidth and increase power consumption due to extensive computing requirements.
Innovation Solution
A neural processor circuit with a neural task manager that switches between task queues based on priority parameters, utilizing neural engines to execute tasks efficiently, including a neural task manager circuit, task queues, and a task arbiter to manage and switch between tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a CPU is used to execute neural network operations, then the system is easy to configure and instantiate different machine learning models, but the CPU bandwidth consumption and power consumption increase significantly
Solution Approach 1:
The patent segments the CPU's neural network processing functions into a dedicated neural processing unit (NPU) with specialized hardware components including neural network parameter storage, operation execution units, and result storage. This segmentation allows the NPU to handle neural network operations independently, reducing CPU bandwidth consumption and power usage while maintaining the ability to configure different models through parameter loading.
2Loss of energy
If a dedicated neural processing unit is introduced to reduce CPU consumption, then power consumption and CPU bandwidth are reduced, but the device complexity increases
Solution Approach 1:
The neural processing unit is designed with universal functionality to execute multiple types of neural network operations (convolutions, activations, pooling) and support various neural network architectures. The NPU can be configured to handle different machine learning models by loading appropriate parameters, making it a multi-functional component that reduces overall system complexity despite adding dedicated hardware.
Solution Approach 2:
The patent implements a hierarchical memory structure where the NPU has local parameter storage nested within the processing unit, which can be supplemented by external memory. This nested architecture allows the NPU to access frequently used parameters quickly from local storage while maintaining the ability to load additional parameters from external memory, reducing memory access complexity and improving performance.
3Productivity
If multiple tasks are queued in the neural processing unit, then task switching capability is improved, but the task management complexity increases
Solution Approach 1:
The patent implements a task queue mechanism where multiple neural network tasks are pre-loaded and organized in the NPU's parameter storage before execution. The task manager pre-configures operation sequences and parameters, allowing the NPU to switch between tasks by simply loading the next pre-prepared task parameters from the queue, reducing the complexity of real-time task management while improving productivity.
Data Source
AI summary
Embodiments relate to managing tasks that when executed by a neural processor circuit instantiates a neural network. A neural task manager circuit within the neural processor circuit can switch between tasks in different task queues. Each task queue is configured to store a reference to a task list of tasks for instantiating a neural network. Each task queue can also be assigned a priority parameter. While the neural processor circuit is executing tasks of a first task list and prior to completion of each task, the neural task manager circuit can switch between task queues according to the priority parameters for execution of tasks of a second task list by the neural processor circuit. The neural processor circuit includes one or more neural engine circuits that are configured to perform neural operations by executing the tasks assigned by the task manager.