Dot Product Engine Crossbar Assignment for Matrix-Vector Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing general-purpose processors, such as CPUs, are inefficient for AI/ML/DL applications due to high memory and computational resource consumption, while GPUs are power-intensive and expensive, necessitating the development of specialized accelerators like dot product engines (DPEs) for efficient matrix-vector multiplication operations.
Innovation Solution
A DPE utilizes memristor crossbars for analog domain operations, with a software development environment and compiler toolchain that optimizes neural network performance by partitioning computation graphs and scheduling MVM operations across parallel compute units, ensuring co-location of weight matrices to minimize data transfer and maximize parallel execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose processors (CPUs) are used for neural network operations, then versatility is maintained, but memory and computational resource consumption increases significantly
Solution Approach 1:
The system segments the computation graph into multiple subgraphs and partitions MVM operations across multiple crossbars. Each crossbar handles specific operations independently, dividing the overall computational workload into manageable segments that can be processed in parallel, thereby reducing the memory and computational burden on any single processing unit.
Solution Approach 2:
The patent introduces a compiler toolchain as an intermediary that automatically partitions the computation graph and schedules MVM operations across available crossbars. This intermediary layer translates high-level neural network definitions into optimized low-level instructions, managing resource allocation without requiring changes to the underlying hardware architecture.
2Power
If GPUs are used for AI/ML/DL applications, then computational power is improved, but power consumption and cost increase
Solution Approach 1:
The patent replaces traditional digital computing mechanics with analog domain operations using memristor crossbars. Matrix-vector multiplication is performed through analog electrical signals and resistive memory elements, eliminating the need for complex digital logic circuits and reducing power consumption while maintaining computational capability.
Solution Approach 2:
The system changes the operational domain from digital to analog, utilizing continuous electrical parameters (voltage, current, resistance) to represent and process neural network weights and activations. This parameter change enables more efficient computation with lower power consumption compared to binary digital operations.
3Productivity
If MVM operations are distributed across multiple crossbars, then parallel execution is improved, but data transfer overhead increases
Solution Approach 1:
The compiler performs preliminary analysis of the computation graph to identify operations that can be co-located on the same crossbar. By pre-scheduling dependent MVM operations on consecutive crossbars and co-locating weight matrices with their corresponding operations, the system minimizes data transfer requirements before execution begins.
Solution Approach 2:
The patent merges related operations and weight matrices onto the same crossbar or consecutive crossbars. By combining operations that share common data dependencies onto adjacent processing units, the system enables efficient data sharing and eliminates redundant data transfers between crossbars during parallel execution.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach significantly improves the efficiency of neural network performance on DPEs by reducing data transfer overheads and optimizing resource utilization, achieving orders of magnitude better performance compared to digital ASICs, especially in large crossbar array sizes.
Implementation Method 1
A DPE is a high-density, power efficient accelerator that utilizes the current accumulation feature of a memristor crossbar
Implementation Method 2
perform MVM operations in the analog domain
Data Source
AI summary
Example implementations relate to assigning dependent matrix-vector multiplication (MVM) operations to consecutive crossbars of a dot product engine (DPE). A method can comprise grouping a first MVM operation of a computation graph with a second MVM operation of the computation graph where the first MVM operation is dependent on a result of the second MVM operation, assigning a first crossbar of a DPE to an operand of the first MVM operation, and assigning a second crossbar of the DPE to an operand of the second MVM operation, wherein the first and second crossbars are consecutive.