Encoder-Decoder Residual Paths for Stable Neural Network Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing encoder-decoder models in machine translation using neural networks face challenges in maintaining learning stability without increasing the number of parameters.

Innovation Solution

Implementing a stack of encoder and decoder layers with residual connections and auxiliary paths that conditionally add outputs based on predetermined conditions, preventing unstable outputs from propagating, thereby stabilizing learning without increasing model parameters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multi-hop attention mechanism is applied to replace multi-head attention, then learning stability is improved, but the number of parameters increases

Engineering Contradiction:

Improvelearning stabilityVSAvoidnumber of parameters

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The residual connection is segmented into two types: standard residual connection and auxiliary residual connection. The auxiliary residual connection is specifically activated in encoder layers 2 and above, creating a differentiated connection structure that stabilizes learning without uniformly increasing parameters across all layers.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The auxiliary residual connection is dynamically activated only in encoder layers 2 and above based on layer depth, rather than being uniformly applied across all encoder layers. This dynamic activation pattern allows the model to gain learning stability benefits while minimizing parameter increases in earlier layers.

Inventive Principle:

Principle #15Dynamics

2Measurement precision

If encoder-decoder model with attention mechanism is used, then translation accuracy is improved, but learning stability deteriorates when learning rate is high or batch size is small

Engineering Contradiction:

Improvetranslation accuracyVSAvoidlearning stability

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The auxiliary residual connection provides an additional feedback pathway that connects encoder layer outputs back to earlier encoder layers. This feedback mechanism helps maintain stable gradients during training, allowing the model to achieve both high translation accuracy and learning stability even at higher learning rates or smaller batch sizes.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The auxiliary residual connection acts as a protective mechanism that is activated in advance in encoder layers 2 and above, cushioning against gradient vanishing or explosion before they can destabilize training. This beforehand cushioning allows the model to tolerate higher learning rates without sacrificing learning stability.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS20260004131A1Learning device, learning method, record medium storing learning program, inference device, inference method, and record medium storing inference program

Publication Date: 2026.01.01 MITSUBISHI ELECTRIC CORP

AI summary

A learning device includes an encoder including N encoder layers and a decoder including M decoder layers. The encoder includes a first path to add a first output as an output of a first encoder layer among the N encoder layers to a first auxiliary residual connection as a residual connection of a second encoder layer that is two or more layers lower than the first encoder layer. The decoder includes a second path to add a second output as an output of a first decoder layer among the M decoder layers to a second auxiliary residual connection as a residual connection of a second decoder layer that is two or more layers lower than the first decoder layer.