Shared Key-Value Attention Mechanism for Neural Network Memory Reduction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Neural networks with attention mechanisms face significant processing and memory resource burdens due to complex operations involving multiple heads and extensive data manipulation, particularly in generating attention information for tasks like text synthesis.

Innovation Solution

Implementing an attention mechanism that generates attention information using head-specific query information and shared key-value (KV) information without caching or generating head-specific key and value information, thereby reducing memory storage and computation requirements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If head-specific key information and head-specific value information are generated and cached in memory for each head in the attention mechanism, then the attention mechanism can perform operations in different representational subspaces with improved accuracy, but the processing and memory resource burden increases significantly

Engineering Contradiction:

Improveattention mechanism accuracyVSAvoidprocessing and memory resource burden

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent merges the key and value information across all heads into shared key information and shared value information. Instead of maintaining separate head-specific key and value matrices, the invention combines them into single shared matrices that are reused by all heads, thereby reducing memory storage requirements and computational overhead while still enabling multi-head attention operations in different representational subspaces

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The shared key information and shared value information serve multiple functions across all attention heads simultaneously. A single shared key matrix and shared value matrix are used by all heads to perform attention operations, making these data structures universal resources that eliminate the need for duplicate head-specific copies, thus reducing overall memory usage and processing complexity

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If multiple attention mechanisms operate in multiple stages with multiple heads each, then the neural network can process information in plural representational subspaces improving model capability, but the data-intensive operations place significant burden on processing resources

Engineering Contradiction:

Improvemodel capability in representational subspacesVSAvoidprocessing resource consumption

Core Design Contradiction:

Adaptability or versatilityVSPower

Solution Approach 1:

The patent combines key information from all heads into a single shared key matrix and value information from all heads into a single shared value matrix. This merging eliminates redundant computations across multiple heads and stages, reducing processing resource consumption while preserving the ability to operate in multiple representational subspaces through the shared structures

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The invention discards the traditional approach of generating and caching head-specific key and value information for each head. Instead, it recovers computational efficiency by using shared key and value information that can be reused across all heads, eliminating wasted processing on redundant operations while maintaining attention mechanism functionality

Inventive Principle:

Principle #34Discarding and recovering

3Speed

If head-specific key information and head-specific value information are cached in memory for each head, then the attention mechanism can efficiently retrieve data for each head, but the amount of information stored in memory increases significantly

Engineering Contradiction:

Improvedata retrieval efficiencyVSAvoidmemory storage requirements

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The patent merges all head-specific key information into a single shared key matrix and all head-specific value information into a single shared value matrix. This consolidation dramatically reduces memory storage requirements from O(n_heads × sequence_length × embedding_dim) to O(sequence_length × embedding_dim), while the shared structures can still be efficiently accessed by all heads for attention computations

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20220318601A1Resource-Efficient Attention in a Neural Network

Publication Date: 2022.10.06 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Computing technology is described herein that provides an attention mechanism, implemented by a neural network, that generates attention information based on head-specific query information and shared key and value (KV) information, without computing head-specific key information and head-specific value information, and without caching the head-specific key information and the head-specific value information in memory. This manner of operation allows the computing technology to make efficient use of processing and memory resources. In some implementations, the attention mechanism is part of decoder of an encoder-decoder system, or a standalone decoder system. In some implementations, the computing technology leverages the attention information to generate synthesized text based on input text.