Tensor Processor Weight Buffer Activation Tensor Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators face performance bottlenecks in edge devices with strict power consumption constraints, particularly when performing inference exercises in real-time applications like AR/VR, due to high computational demands of convolutional and self-attention layers.
Innovation Solution
A computing system with tensor processor clusters and an instruction master that performs batch matrix multiplication between activation tensors, utilizing a weight DMA to split and reorder activation tensors for efficient processing, allowing reuse of weight matrices and transposing tensors to optimize computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing ML accelerators use high compute parallelism and optimized data orchestration, then processing speed of convolutional and self-attention layers is improved, but power consumption increases making them unsuitable for edge devices with strict power constraints
Solution Approach 1:
The patent segments the second activation tensor into two blocks and further divides each block into MSB and LSB tiles. This segmentation allows the weight matrix to be reused across multiple tensor blocks without reloading from memory, reducing power consumption while maintaining processing throughput through parallel tile processing
Solution Approach 2:
The weight matrix is pre-loaded into the weight buffer before processing multiple activation tensor blocks. This preliminary action eliminates repeated memory access operations during batch processing, significantly reducing power consumption while maintaining high processing speed through efficient data reuse
2Adaptability or versatility
If ML accelerators are designed for CNN/MLP workloads with MAC/GEMM operators, then offline-trained weight processing is optimized, but activation-to-activation multiplication for attention mechanisms cannot be performed
Solution Approach 1:
The patent modifies the weight DMA and weight buffer to handle activation tensors universally, allowing the same hardware infrastructure to process both traditional weight matrices and activation tensors. This enables attention mechanism support while maintaining efficiency through the established MAC unit architecture
Solution Approach 2:
The system changes the interpretation of the weight buffer from storing only weight matrices to storing any 2D tensor (weights or activations). The DMA controllers are configured to load data with different semantic meanings but identical data formats, enabling workload versatility without sacrificing processing efficiency
3Productivity
If the second activation tensor is stored in activation buffers of N tensor processor units, then distributed processing is enabled, but the weight DMA must collect and redistribute data increasing complexity
Solution Approach 1:
The weight DMA acts as an intermediary that collects scattered tensor blocks from multiple activation buffers, reorganizes them into a contiguous structure with proper MSB/LSB tiling, and redistributes them to the weight buffer. This centralized mediation simplifies the complexity by providing a single point of data orchestration rather than requiring coordinated access across all N tensor processor units
Data Source
AI summary
In one embodiment, a method includes, determining that a bmm operation between a first activation tensor and a second activation tensor needs to be performed, collecting the second activation tensor in two blocks from activation buffers of N tensor processor units, splitting each of the two blocks of the second activation tensor into an MSB tile and an LSB tile, loading the second activation tensor to weight buffers of the N tensor processor units by filling a first entry of each weight buffer of each of the N tensor processor units with contents of the MSB tiles of the two blocks and filling a second entry of the weight buffer with contents of the LSB tiles of the two blocks, and generating a bmm result using the first activation tensor distributed in the activation buffers and the second activation tensor in the weight buffers.