Neural Network Layer Scheduling Across Cores for Adaptive Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently mapping neural network layers to physical neural cores in inference processing units, leading to suboptimal performance and resource utilization.
Innovation Solution
A scheduler is employed to map neural network layers to an array of neural cores, utilizing a scheme library and graph sequencer to determine precedence relationships and optimize data movement, while ensuring efficient memory management and parallel computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network layers are mapped to physical neural cores using existing technologies, then the mapping can be completed, but the computational efficiency and resource utilization are suboptimal
Solution Approach 1:
The system dynamically reconfigures parallelism during runtime by adjusting the number of active neural cores based on batch size and network configuration. The scheduler adapts the mapping strategy in real-time to optimize computational efficiency for different workloads, transitioning between various parallelism configurations without requiring system restart or manual intervention.
Solution Approach 2:
The patent changes key parameters including batch size, number of active cores, and parallelism degree to optimize performance. By varying these parameters based on workload characteristics, the system achieves optimal resource utilization and computational efficiency for different neural network inference scenarios.
2Productivity
If more neural cores are activated to increase throughput, then processing capacity improves, but energy consumption increases
Solution Approach 1:
The system dynamically adjusts the number of active neural cores based on the inference workload and batch size. When batch size is small, fewer cores are activated to reduce energy consumption. When batch size increases, more cores are activated to maintain throughput efficiency, achieving optimal energy-performance tradeoff adaptively.
Solution Approach 2:
The patent varies the activation level of neural cores as a controllable parameter. By changing the number of active cores from 1 to N based on workload demands, the system optimizes the balance between throughput and energy consumption, activating only the necessary computational resources for each inference task.
3Adaptability or versatility
If static mapping is used to simplify scheduling, then implementation is easier, but adaptability to different workloads is reduced
Solution Approach 1:
The scheduler transitions from static to dynamic mapping by continuously monitoring workload characteristics and adjusting the neural core assignment accordingly. Different mapping strategies are applied based on batch size, network architecture, and computational requirements, enabling the system to adapt to diverse workloads while managing complexity through automated decision-making.
Solution Approach 2:
The patent implements a universal scheduling framework that handles multiple workload types and network configurations through a single adaptive system. The same scheduler infrastructure supports various batch sizes, network architectures, and parallelism configurations, eliminating the need for workload-specific mapping implementations.
4Productivity
If memory access is optimized for sequential processing, then memory efficiency improves, but parallel computation performance deteriorates
Solution Approach 1:
The patent segments the neural network computation into independent parallel tasks that can be distributed across multiple cores. By dividing the weight tensors and activation data into partitions, each core processes a specific segment simultaneously, enabling parallel computation while maintaining efficient memory access patterns through localized data access within each segment.
Solution Approach 2:
The system transitions from sequential to parallel processing by adding the temporal dimension of simultaneous execution. Multiple neural cores operate in parallel on different data partitions, transforming the computation from a single-threaded sequential process to a multi-threaded parallel process, thereby improving both performance and memory utilization efficiency.
Data Source
AI summary
Mapping of neural network layers to physical neural cores is provided. In various embodiments, a neural network description describing a plurality of neural network layers is read. Each of the plurality of neural network layers has an associated weight tensor, input tensor, and output tensor. A plurality of precedence relationships among the plurality of neural network layers is determined. The weight tensor, input tensor, and output tensor of each of the plurality of neural network layers are mapped onto an array of neural cores.