Neural Network Data Partitioning to Reduce Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
When multiple neural processors attempt to accelerate a neural network operation, redundant data between processors increases memory bandwidth, and synchronization and data copying lead to performance degradation, with conventional methods failing to consider optimal division and scheduling for multiple processors.
Innovation Solution
A method and apparatus that involve receiving data on a neural network layer, generating partitions and tiles based on partitioning policies and data size, and scheduling operations to minimize redundancy and synchronization, optimizing data assignment and operation order across cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data are divided among multiple neural processors, then processing speed is improved, but memory bandwidth increases due to redundant data
Solution Approach 1:
The patent segments data into partitions and further divides partitions into tiles, creating a hierarchical partitioning structure. This allows different levels of data sharing: partitions can be shared across cores while tiles within a partition are processed independently, reducing redundant data transmission while maintaining parallel processing speed benefits.
Solution Approach 2:
The patent applies different sharing strategies to different data granularities: partitions are designed to be shared among cores to reduce memory bandwidth, while tiles are processed locally without sharing. This local quality differentiation optimizes both memory bandwidth utilization and processing speed by avoiding redundant transmissions at the tile level.
2Productivity
If multiple neural processors perform divided operations, then processing throughput is improved, but synchronization frequency increases causing performance degradation
Solution Approach 1:
By segmenting data into partitions that can be independently processed by different cores, the patent enables parallel processing throughput improvement while minimizing synchronization needs. Each core processes its assigned partitions independently, reducing the frequency of synchronization operations.
Solution Approach 2:
The scheduler performs preliminary actions by pre-calculating optimal tile schedules and data sharing strategies before execution begins. This allows the system to proceed with minimal interruptions for synchronization, maintaining high throughput while reducing time loss from frequent sync operations.
3Device complexity
If conventional single-processor methods are used, then implementation simplicity is maintained, but core utilization and data copying efficiency are reduced
Solution Approach 1:
The patent segments the neural network operations into partitions and tiles that can be distributed across multiple cores. This segmentation enables multi-core utilization while maintaining a manageable implementation structure through hierarchical organization and standardized scheduling interfaces.
Solution Approach 2:
The scheduler component performs multiple functions: it manages partition assignment, determines data sharing strategies, schedules tile execution, and coordinates core utilization. This multi-functionality consolidates complexity into a single manageable component while improving overall productivity through optimized resource utilization.
Data Source
AI summary
A method and apparatus for scheduling a neural network operation. The method includes receiving data on a layer of a neural network, generating partitions to be assigned to cores by dividing the data, generating tiles by dividing the partitions, and scheduling an operation order of the tiles based on whether the data are shared between the cores.