Computation Engine Matrix Vector Mode Parallel Processing

Solution Overview

Problem

Modern computing systems face performance and power efficiency issues when performing massive vector multiplications and outer product operations, particularly in battery-powered systems where general-purpose CPUs with vector instructions exhibit low performance and high power consumption.

Innovation Solution

A computation engine is designed to perform vector multiplications and outer product operations efficiently by specifying a matrix or vector mode, allowing for parallel computations and flexible operand locations, and supporting instructions like fused multiply-add and multiply-accumulate operations to enhance performance and reduce power consumption.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If general-purpose CPU with vector instructions is used to perform vector multiplications, then the system can handle diverse workloads, but the performance is very low and power consumption is very high

Engineering Contradiction:

Improveworkload handling capabilityVSAvoidvector multiplication performance

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The computation engine is segmented into dedicated functional units including multiple vector processing units (VPUs) and matrix processing units (MPUs), each specialized for specific operation types. This segmentation allows simultaneous execution of different operation types in parallel, achieving high performance for vector multiplications while maintaining the ability to handle diverse workloads through configurable processing units.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The computation engine acts as an intermediary between the general-purpose CPU and memory systems, offloading computationally intensive vector and matrix operations from the CPU. The engine includes dedicated input memories (X and Y) and result memories (Z) that serve as intermediaries for data exchange, reducing memory access latency and improving overall system performance for machine learning workloads.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If general-purpose CPU with vector instructions is used to perform vector multiplications, then the system can handle diverse workloads, but the power consumption is very high

Engineering Contradiction:

Improveworkload handling capabilityVSAvoidpower consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The computation engine is segmented into dedicated functional units including multiple vector processing units (VPUs) and matrix processing units (MPUs), each specialized for specific operation types. This segmentation allows simultaneous execution of different operation types in parallel, achieving high performance for vector multiplications while maintaining the ability to handle diverse workloads through configurable processing units.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The computation engine acts as an intermediary between the general-purpose CPU and memory systems, offloading computationally intensive vector and matrix operations from the CPU. The engine includes dedicated input memories (X and Y) and result memories (Z) that serve as intermediaries for data exchange, reducing memory access latency and improving overall system performance for machine learning workloads.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Productivity

If computation engine performs numerous computations in parallel, then the performance is high, but the device complexity increases

Engineering Contradiction:

Improvecomputation performanceVSAvoidengine structure complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The computation engine employs universal processing units that can operate in multiple modes. The vector processing units (VPUs) can perform both vector operations and matrix operations depending on configuration, and the matrix processing units (MPUs) can handle different matrix dimension combinations. This multi-functionality allows the engine to achieve high performance through parallel processing while reducing the number of specialized units needed, thereby controlling complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The engine handles different operation types by adding a configuration dimension rather than creating separate hardware paths. Instructions include mode bits that switch the processing units between vector and matrix operation modes, allowing the same physical units to serve multiple purposes. This dimensional approach to configurability enables high performance parallel processing without proportionally increasing device complexity.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11042373B2Computation engine that operates in matrix and vector modes

Publication Date: 2021.06.22 APPLE INC

AI summary

In an embodiment, a computation engine is configured to perform vector multiplications, producing either vector results or outer product (matrix) results. The instructions provided to the computation engine specify a matrix mode or a vector mode for the instructions. The computation engine performs the specified operation. The computation engine may perform numerous computations in parallel, in an embodiment. In an embodiment, the instructions may also specify an offset with the input memories, providing additional flexibility in the location of operands. More particularly, the computation engine may be configured to perform numerous multiplication operations in parallel and to accumulate results in a result memory, performing multiply-accumulate operations for each matrix/vector element in the targeted locations of the output memory.