D Flip-Flop Buffer for Tensor Transpose Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning accelerators face performance bottlenecks in edge devices with strict power consumption constraints, particularly in real-time inference exercises for applications like virtual reality, augmented reality, and mobile devices, due to high computational demands of convolutional and self-attention layers.
Innovation Solution
A computing system utilizing a single D Flip-Flop memory buffer to efficiently perform tensor transposes, allowing simultaneous reading and writing operations, thereby achieving the throughput of double buffering without the need for two buffers, optimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single SRAM memory buffer is used for tensor transpose, then memory usage is reduced, but bandwidth utilization deteriorates due to sequential access limitations
Solution Approach 1:
The patent changes the fundamental parameter of memory access by using D Flip-Flop elements that enable simultaneous read and write operations on the same memory location, transforming the sequential access model into a parallel access model. This allows the single buffer to achieve the throughput of double buffering by allowing overlapping of read and write operations without conflict.
2Productivity
If double buffering is used for tensor transpose, then bandwidth utilization is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent merges the functionality of two separate buffers into a single buffer by using D Flip-Flop elements that can simultaneously handle read and write operations. The merging is achieved through the inherent dual-port capability of the D Flip-Flop structure, which allows one buffer to serve the purposes of two buffers while reducing overall device complexity and resource consumption.
Solution Approach 2:
The single D Flip-Flop buffer performs multiple functions that would traditionally require two separate buffers: it can simultaneously read data in one order while writing data in another order, and it can switch between different access patterns. This multi-functionality eliminates the need for separate buffers while maintaining the performance benefits of double buffering.
3Ease of manufacture
If traditional memory buffering is used, then implementation is simple, but power consumption increases in edge devices
Solution Approach 1:
The patent uses D Flip-Flop elements as a copy mechanism that enables efficient data transfer and transformation. The flip-flops create temporary copies of data in a way that enables simultaneous read/write operations, achieving high efficiency with reduced power consumption compared to traditional SRAM buffering approaches.
Data Source
AI summary
In one embodiment, a method includes, at each iteration i among N iterations of a first loop, reading first data corresponding to row i of a first tensor from a first source memory, reading second data from column i of the transpose buffer, writing the first data to column i of the transpose buffer, and causing the second data to be written to row i of a second tensor at a first destination memory and, at each iteration j among N iterations of a second loop, reading third data corresponding to row j of a third tensor from a second source memory, reading fourth data from row j of the transpose buffer, writing the third data to row j of the transpose buffer, and causing the fourth data to be written to row j of a fourth tensor at a second destination memory.