Outer Product Matrix-Vector Multiplication Accelerator

Solution Overview

Problem

Current AI semiconductor accelerators optimized for matrix-matrix multiplication operations face reduced speed and utilization in matrix-vector multiplication due to memory bandwidth limitations, leading to inefficient hardware usage and reduced performance in next-generation neural networks like transformer architectures.

Innovation Solution

An outer product-based matrix-vector multiplication operation apparatus and method that reuses the existing matrix-matrix multiplication hardware structure, utilizing internal calculators for MAC operations, data transmission paths, and multiplexers to enhance memory interface utilization and calculator efficiency by processing multiple operands simultaneously, including half-precision data and exponent-bias floating-point types.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If memory bandwidth is increased to accelerate matrix-vector multiplication operations, then the speed of matrix-vector multiplication is improved, but hardware area and system complexity increase

Engineering Contradiction:

Improvespeed of matrix-vector multiplicationVSAvoidhardware area and system complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent makes the matrix-matrix multiplication calculator perform both matrix-matrix multiplication and matrix-vector multiplication operations by reconfiguring data flow paths. The same calculator hardware processes different operation types, eliminating the need for separate dedicated hardware for matrix-vector multiplication and avoiding increased system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces an outer product calculator as an intermediary component that generates outer products from input vectors, which are then processed by the existing matrix-matrix multiplication calculator. This intermediary approach enables matrix-vector multiplication acceleration without directly modifying the core calculator hardware, thus avoiding increased hardware area.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If memory bandwidth is increased to enhance accelerator utilization, then matrix-vector multiplication speed is improved, but memory bandwidth utilization in matrix-matrix multiplication deteriorates

Engineering Contradiction:

Improveaccelerator utilizationVSAvoidmemory bandwidth utilization efficiency

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent implements dynamic data flow configuration where the same hardware infrastructure adapts its data transmission paths based on the operation type. For matrix-vector multiplication, data flows through the outer product calculator and selected calculator units; for matrix-matrix multiplication, the original data flow is used. This dynamic reconfiguration allows full memory bandwidth utilization in both operation modes without waste.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent changes operational parameters by using half-precision data formats (FP16) instead of full-precision formats, reducing data width and enabling more data to be processed simultaneously through the same bandwidth. This parameter change allows the system to process more operands at the same memory bandwidth, improving accelerator utilization without increasing actual memory bandwidth requirements.

Inventive Principle:

Principle #35Parameter changes

3Speed

If dedicated hardware structure is created for matrix-vector multiplication, then operation speed is improved, but hardware area and complexity increase

Engineering Contradiction:

Improvematrix-vector multiplication speedVSAvoidhardware area

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent makes the matrix-matrix multiplication calculator perform both matrix-matrix multiplication and matrix-vector multiplication operations by reconfiguring data flow paths. The same calculator hardware processes different operation types, eliminating the need for separate dedicated hardware for matrix-vector multiplication and avoiding increased system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces an outer product calculator as an intermediary component that generates outer products from input vectors, which are then processed by the existing matrix-matrix multiplication calculator. This intermediary approach enables matrix-vector multiplication acceleration without directly modifying the core calculator hardware, thus avoiding increased hardware area.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20240362297A1Outer product-based matrix-vector multiplication operation apparatus for accelerating vector operation and method using the same

Publication Date: 2024.10.31 ELECTRONICS & TELECOMM RES INST

AI summary

Disclosed herein are an outer product-based matrix-vector multiplication operation apparatus and a method using the same. The outer product-based matrix-vector multiplication operation apparatus includes internal calculators, each configured to generate an accumulated value by performing a Multiply-Accumulation (MAC) operation, an internal data transmission path configured to simultaneously provide a vector element to two or more internal calculators, and at least one multiplexer configured to select any one of the vector element and a vector of a matrix, wherein a first input port of each of the internal calculators is connected to one of vectors of the matrix and a second input port of each of the internal calculators is connected to the vector element.