Coupled Matrix Compute Engine With Inline Buffer Thread Packing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing matrix processing engines face challenges in achieving high performance per watt and per unit area, flexibility in mapping diverse matrix operations, and efficient resource utilization due to inefficient data movements and synchronization issues, particularly in AI/CV applications.
Innovation Solution
A matrix processing engine that combines a dense matrix compute circuit with a programmable scalar computing core, utilizing a single-instruction-multiple-data (SIMD) and single-instruction-single-data (SISD) architecture, with in-line register buffers and thread packet generation to tightly couple vector and scalar operations, enabling efficient data sharing and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a unified architecture combines matrix multiplication and scalar operations with fine-grained data coupling, then operational flexibility and integration are improved, but data movement overhead and synchronization complexity increase
Solution Approach 1:
The processor is segmented into distinct matrix processing units and scalar processing units, each handling specific operations. This segmentation allows independent optimization of each unit while maintaining controlled interfaces through register files, reducing the synchronization complexity that would arise from a fully unified architecture.
Solution Approach 2:
Register files serve as intermediary structures between matrix processing units and scalar processing units. These register files buffer data and instructions, mediating the fine-grained data coupling required for operational flexibility while absorbing the synchronization complexity within the register interface layer rather than propagating it throughout the entire system.
2Productivity
If vector and scalar operations are tightly coupled with frequent data movements, then computational efficiency is improved, but data transfer overhead and operand latency increase
Solution Approach 1:
The architecture performs preliminary actions by pre-loading operands into register files before they are needed by processing units. This allows data to be staged and prepared in advance, reducing operand latency when computations need to proceed at high speed.
Solution Approach 2:
The patent introduces a temporal dimension to data flow management through multi-cycle instruction execution and staged register file operations. Instead of simple sequential data movement, the system uses multi-dimensional data paths that allow operands to be prepared, buffered, and delivered across multiple time cycles, reducing critical path latency.
3Productivity
If custom accelerator designs use fixed operations and dedicated internal data paths, then performance for specific workloads is improved, but adaptability to diverse matrix and DNN functions decreases
Solution Approach 1:
The processor employs universal processing units that can execute multiple operation types. The matrix processing units can perform different matrix operations (multiplication, accumulation, transposition) and the scalar processing units can handle various scalar functions (activation functions, normalization), providing multi-functionality without requiring separate dedicated hardware for each operation.
Solution Approach 2:
The architecture introduces dynamic configurability through programmable control logic that can reconfigure data paths and operation modes at runtime. This allows the same hardware structure to adapt its behavior for different workloads, maintaining high performance while supporting diverse matrix and DNN functions.
4Adaptability or versatility
If general-purpose solutions are used for matrix processing, then adaptability to diverse operations is improved, but chip-area cost and resource utilization efficiency worsen
Solution Approach 1:
The patent applies local quality by giving different parts of the processor specialized characteristics optimized for their specific functions. Matrix processing units have structures optimized for parallel matrix operations, while scalar processing units are optimized for sequential scalar computations. This localized optimization achieves high adaptability without requiring the entire chip to be over-provisioned for the most demanding operations.
Solution Approach 2:
The architecture merges matrix processing capabilities and scalar processing capabilities into a single integrated processor. By combining these previously separate functions into one unified device with shared resources (such as register files and control logic), the system achieves diverse operational adaptability while reducing the total chip area compared to having separate dedicated processors for each function.
Data Source
AI summary
A matrix processing engine is provided for efficient matrix computation performed by a dense matrix compute circuit (performing SIMD operations) and a scalar computing core (performing SISD operations). These two processing components operate together to produce output data tiles by feeding results of the dense SIMD operations to the scalar computing core using thread packing and an in-line buffer for accumulating and packing the dense result data. This permits the scalar computing to spawn threads to operate on the dense results as available and without requiring partial or intermediate data read/writes between the dense and scalar computations.