Encoder-Decoder Residual Paths for Stable Deep Learning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing encoder-decoder models in machine learning face challenges in maintaining learning stability without increasing the number of parameters.

Innovation Solution

Incorporating a stack of encoder and decoder layers with residual connections and auxiliary paths to connect outputs of specific layers, allowing for stable parameter updates without increasing the model's parameter count.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multi-hop attention mechanism is applied to replace multi-head attention, then learning stability is improved, but the number of parameters increases

Engineering Contradiction:

Improvelearning stabilityVSAvoidnumber of parameters

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent segments the residual connection path by introducing auxiliary residual connections that skip specific layers (e.g., skipping 2 or more layers). This segmentation allows gradient flow through alternative paths without requiring the complex multi-hop attention structure, thereby maintaining learning stability while reducing parameter count.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces auxiliary residual connections as intermediary pathways that facilitate gradient flow between encoder and decoder layers. These auxiliary connections act as mediators that stabilize learning by providing direct gradient routes without requiring the parameter-intensive multi-hop attention mechanism.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If encoder-decoder model depth is increased, then translation accuracy is improved, but gradient vanishing problem worsens

Engineering Contradiction:

Improvetranslation accuracyVSAvoidgradient vanishing

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent applies preliminary action by pre-establishing auxiliary residual connection paths before training begins. These pre-configured pathways ensure that gradient flow is maintained throughout deep networks from the start of training, preventing gradient vanishing before it can occur during the training process.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent adds another dimension to the network architecture by introducing auxiliary residual connections that operate in parallel with the main forward-propagation path. This additional dimensional pathway for gradient flow allows the network to maintain stability at greater depths without being constrained by the limitations of the primary connection path.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentEP4685698A1Learning device, learning method, learning program, inference device, inference method, and inference program

Publication Date: 2026.01.28 MITSUBISHI ELECTRIC CORP

Figure 1
Figure 2~3
Figure 4~5

AI summary

A learning device (11) includes an encoder (11e) including N encoder layers and a decoder (11d) including M decoder layers. The encoder (11e) includes a first path (22e) to add a first output as an output of a first encoder layer (11e_n, 1 ≤ n ≤ N - 2) among the N encoder layers (11e_1, ..., 11e_N) to a first auxiliary residual connection (21e) as a residual connection of a second encoder layer (11e_n+α, α ≥ 2) that is two or more layers lower than the first encoder layer (11e_n). The decoder (11d) includes a second path (22d) to add a second output as an output of a first decoder layer (11d_m, 1 ≤ m ≤ M - 2) among the M decoder layers (11d_1, ..., 11d_M) to a second auxiliary residual connection (21d) as a residual connection of a second decoder layer (11d_n+β, β ≥ 2) that is two or more layers lower than the first decoder layer (11d_m).