Software-Defined Buffer Transposer for Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic circuits face performance bottlenecks in massively parallel general matrix multiplication due to slow data reformatting on host CPUs and inefficient data streaming, which limits the effective double data rate (DDR) memory throughput and requires separate buffer circuits or complex data reordering.
Innovation Solution
A software-defined buffer/transposer implemented in a programmable integrated circuit (IC) that simultaneously buffers and reformats input data streams, achieving linear DDR addressing by using hardware logic to reorganize data into compute-engine-friendly formats, thereby eliminating the need for separate buffering and reordering circuits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If separate buffer circuits and reordering circuits are used, then data buffering and reformatting can be performed, but device complexity increases and resource usage increases
Solution Approach 1:
The patent combines the buffer circuit and reordering circuit into a single integrated buffer/transposer unit. The buffer memory is shared between buffering operations and reformatting operations, eliminating the need for separate circuits. The control logic unifiedly manages both data buffering and row-by-row reformatting to column-major order, reducing overall device complexity while maintaining full functionality.
Solution Approach 2:
The buffer memory serves multiple functions simultaneously: it acts as a data buffer for incoming matrix elements and as a reordering structure for transposing data from row-major to column-major format. The single circuit structure performs both buffering and reformatting tasks, making it a multi-functional unit that eliminates redundant hardware resources.
2Ease of operation
If data reformatting is performed on host CPU, then data can be reformatted, but processing speed decreases due to slow CPU reformatting
Solution Approach 1:
The patent replaces the software-based reformatting process on the host CPU with a hardware-based reformatting circuit. The electronic circuit performs data reordering from row-major to column-major format through dedicated hardware logic and memory access patterns, achieving significantly higher speeds compared to software execution on a general-purpose processor.
Solution Approach 2:
The buffer/transposer circuit performs reformatting operations in advance, converting data to column-major order before it reaches the compute engine. This preliminary hardware-based reformatting eliminates the need for slower software-based reformatting later in the processing pipeline, ensuring data is ready in the correct format for efficient matrix multiplication operations.
3Productivity
If efficient data streaming is implemented, then DDR memory throughput increases, but requires complex data reordering logic
Solution Approach 1:
The patent merges the buffering function and reordering function into a single integrated unit. The buffer memory is used for both purposes: storing incoming data streams and reorganizing data into column-major format. This combination eliminates the need for separate complex reordering circuits while maintaining efficient data streaming and maximizing DDR memory throughput.
Solution Approach 2:
The buffer/transposer circuit performs self-service by automatically reformatting data as it is buffered. The control logic inherently handles the row-major to column-major conversion during the buffering process itself, without requiring additional separate reordering operations. This self-service approach simplifies the overall system while enabling efficient data streaming.
Data Source
AI summary
Methods and apparatus are described for simultaneously buffering and reformatting (e.g., transposing) a matrix for high-speed data streaming in general matrix multiplication (GEMM), which may be implemented by a programmable integrated circuit (IC). Examples of the present disclosure increase the effective double data rate (DDR) memory throughput for streaming data into GEMM digital signal processing (DSP) engine multifold, as well as eliminate slow data reformatting on a host central processing unit (CPU). This may be accomplished through software-defined (e.g., C++) data structures and access patterns that result in hardware logic that simultaneously buffers and reorganizes the data to achieve linear DDR addressing.