Neural Network Accelerator Tensor Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network-based computing systems face challenges in efficiently processing large tensors due to increased compile time and potential stalls in tensor operations, particularly when handling partitioned tensors.
Innovation Solution
The approach involves selecting a set of partitioned tensors as a working set, allocating appropriate memory for it, and defining operations through a compute graph. Each operation is executed on a core of the neural network accelerator, with output working sets being directly passed to subsequent operations to prevent stalls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If partitioned tensors are processed sequentially through multiple operations, then computational accuracy is maintained, but stalls occur and productivity decreases
Solution Approach 1:
The patent segments the large tensor processing task into multiple smaller partitioned tensors, each handled by dedicated compute nodes. This segmentation allows parallel processing of different partitions while maintaining computational accuracy through controlled data flow between partitions, thereby eliminating stalls and improving overall productivity.
Solution Approach 2:
The patent introduces a new dimension of parallelism by distributing partitioned tensors across multiple compute nodes simultaneously. Instead of sequential processing, the system processes multiple tensor partitions in parallel across different spatial dimensions of the compute architecture, maintaining accuracy through coordinated data exchange while dramatically improving throughput.
2Adaptability or versatility
If memory is allocated for large tensors, then higher dimension tensors can be processed, but compile time increases
Solution Approach 1:
The patent segments large high-dimensional tensors into smaller partitioned tensors that can be processed independently. This segmentation reduces the compile-time memory allocation overhead for each individual operation while maintaining the capability to handle high-dimensional data through coordinated processing of multiple partitions.
Solution Approach 2:
The patent performs preliminary partitioning of tensors before processing, organizing data into manageable segments in advance. This preliminary action reduces the complexity of memory allocation during compilation by pre-establishing the data structure, allowing high-dimensional tensors to be handled without excessive compile-time overhead.
3Productivity
If compute nodes are dedicated to specific operations, then utilization efficiency improves, but system complexity increases
Solution Approach 1:
The patent implements a universal compute node architecture where each compute node can handle multiple tensor partitions and operations. This multi-functionality improves utilization efficiency by allowing flexible allocation of compute resources to different tasks while avoiding the complexity of dedicated specialized hardware for each operation type.
Data Source
AI summary
Examples of performing tensor operations by a neural network-based computing system, are described. In an example, a first output working set generated by a first operation, wherein the first output working set is a set of processed partitioned tensors, is obtained. The first output working set is then copied to the output working set, for retrieving by the second operation.