Streaming GEMV Circuit With Post-Processing for Low-Latency AI Compute

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing hardware accelerators inefficiencies in performing generalized matrix-vector multiplication (GEMV) operations due to latency and unnecessary computations when using generalized matrix-matrix multiplication (GEMM) circuits, particularly in loading input vectors and matrices and processing zero-valued elements.

Innovation Solution

A hardware accelerator with a specialized GEMV circuit that performs computations through streaming iterations, avoiding buffer loading and incorporating a post-processing circuit for efficient vector operations, and a control processor to determine optimal circuit usage based on operation thresholds.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If GEMM circuits are used to perform GEMV operations, then existing hardware accelerators can perform matrix operations, but latency increases and unnecessary computations occur due to buffer loading and processing of zero-valued elements

Engineering Contradiction:

ImproveGEMV operation efficiencyVSAvoidlatency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent extracts the GEMV operation functionality from the general-purpose GEMM circuit by introducing a specialized GEMV circuit. This dedicated circuit is designed specifically for matrix-vector multiplication, eliminating the overhead of buffer loading and zero-element processing that plagues general GEMM circuits when performing GEMV operations.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies local quality by creating a specialized GEMV circuit with specific architectural features optimized for matrix-vector multiplication. This includes streamlined data paths and computation units tailored for GEMV operations, rather than using a generic GEMM circuit architecture that must handle all matrix operation types.

Inventive Principle:

Principle #3Local quality

2Productivity

If GEMM circuits are used to perform GEMV operations, then existing hardware accelerators can perform matrix operations, but unnecessary computations occur due to processing of zero-valued elements

Engineering Contradiction:

ImproveGEMV operation efficiencyVSAvoidenergy consumption

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent applies partial action by implementing a specialized GEMV circuit that performs only the necessary computations for matrix-vector multiplication, without the overhead of handling general matrix-matrix operations. This avoids the excessive computation of zero-valued elements that occurs when using full GEMM circuits for GEMV tasks.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If a specialized GEMV circuit is introduced, then GEMV operation efficiency improves, but device complexity increases

Engineering Contradiction:

ImproveGEMV operation efficiencyVSAvoidhardware accelerator architecture

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent applies universality by designing a hardware accelerator that can perform both GEMM and GEMV operations through its circuit architecture. The control circuitry determines whether to route GEMM operations to the GEMM circuit or GEMV operations to the GEMV circuit, allowing a single device to handle multiple operation types efficiently.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent applies dynamics by implementing a control mechanism that dynamically selects between GEMM and GEMV circuits based on the operation type. This dynamic routing allows the system to adapt its computational path in real-time, optimizing performance for the specific operation being performed while managing hardware resource utilization.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20260044573A1Hardware accelerator with generalized matrix-vector multiplication and post-processing circuits

Publication Date: 2026.02.12 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A computing device including a hardware accelerator. The hardware accelerator includes a generalized matrix-vector multiplication (GEMV) circuit configured to compute a product vector over a plurality of streaming iterations. At each of the streaming iterations, the GEMV circuit receives an input vector element and an input matrix row. The GEMV circuit multiplies the input vector element by input matrix elements included in the input matrix row to obtain an intermediate product row. The GEMV circuit adds the intermediate product row to a current-iteration row sum. The product vector is equal to the current-iteration row sum computed in a final streaming iteration. The GEMV circuit transmits the product vector as a streaming output to a post-processing circuit included in the hardware accelerator. The post-processing circuit performs a vector processing operation on the product vector to compute vector processing result, and outputs the vector processing result.