Attention Neural Network Linear Units for Sequence Accuracy

Solution Overview

Problem

Conventional attention-based neural networks face limitations in processing input sequences and generating output sequences with accuracy, particularly when unsupervised pre-training is used to enhance performance with limited training data.

Innovation Solution

Implementing a feed-forward sub-layer in attention neural networks that computes an element-wise product between vectors derived from two different linear transformations, replacing the conventional single linear transformation followed by an activation function.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If a single linear transformation followed by an activation function is used in the feed-forward sub-layer, then the device complexity is reduced, but the manufacturing precision (performance accuracy) deteriorates

Engineering Contradiction:

Improvefeed-forward sub-layer structureVSAvoidsequence processing accuracy

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The feed-forward sub-layer is segmented into multiple parallel linear transformations instead of a single transformation. Each transformation processes the input through different learned parameters, and their outputs are combined via element-wise multiplication. This segmentation allows the network to capture multiple feature representations simultaneously, improving sequence processing accuracy while maintaining computational efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The feed-forward sub-layer uses a composite structure combining multiple linear transformations with element-wise multiplication. This composite approach integrates different learned transformations to create a more powerful feature extraction mechanism, analogous to using composite materials to achieve superior properties compared to single materials.

Inventive Principle:

Principle #40Composite materials

2Ease of manufacture

If conventional attention-based neural networks are used, then the ease of manufacture is improved, but the reliability (performance on downstream tasks) deteriorates

Engineering Contradiction:

Improvenetwork implementationVSAvoiddownstream task performance

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The network architecture changes the parameters of the feed-forward sub-layer by introducing multiple linear transformations with different weight matrices and bias vectors. These parameter changes enable the network to learn more diverse feature representations, significantly improving reliability on downstream tasks while maintaining the same basic attention mechanism structure.

Inventive Principle:

Principle #35Parameter changes

3Manufacturing precision

If a feed-forward sub-layer computing element-wise product of two linear transformations is used, then the manufacturing precision (accuracy) is improved, but the device complexity increases

Engineering Contradiction:

Improveoutput sequence accuracyVSAvoidfeed-forward sub-layer structure

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The feed-forward sub-layer is segmented into multiple parallel linear transformations instead of a single transformation. Each transformation processes the input through different learned parameters, and their outputs are combined via element-wise multiplication. This segmentation allows the network to capture multiple feature representations simultaneously, improving sequence processing accuracy while maintaining computational efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The outputs of multiple linear transformations are merged through element-wise multiplication. This combining operation integrates the features learned by different transformations in a computationally efficient manner, achieving improved accuracy without proportionally increasing complexity.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12608616B2Attention neural networks with linear units

Publication Date: 2026.04.21 GOOGLE LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer that applies an element-wise multiplication between two vectors generated as a result of two different linear transformations performed on the same attended layer input.