Reversible Duplex Neural Network for Speech Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech-to-speech (S2S) neural network models are less effective in duplex translations, where speech needs to be translated in both directions, especially in live dialogues between speakers using different languages, due to suboptimal performance and low efficiency of current techniques.
Innovation Solution
A reversible duplex model with a mirrored neural network architecture, incorporating conformer blocks with self-attention, convolutional, and feed-forward modules, and employing diffusion training techniques to achieve cycle consistency in translations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing S2S neural network models are used for duplex translation, then translation capability is provided, but performance and efficiency are suboptimal
Solution Approach 1:
The model segments the translation task into distinct directional pathways (forward and reverse) with specialized encoder-decoder pairs for each direction. This segmentation allows each pathway to be optimized independently for its specific language pair, improving both performance and efficiency in duplex translation scenarios.
Solution Approach 2:
The duplex translation model implements a universal architecture that handles multiple translation directions simultaneously through shared components (embedding layers, attention mechanisms, normalization layers). This multi-functionality enables the system to perform both forward and reverse translations with a single model, improving efficiency while maintaining performance through parameter sharing and joint training.
2Reliability
If separate models are deployed for each translation direction, then translation capability is maintained, but system complexity increases
Solution Approach 1:
The patent merges multiple translation models into a single unified duplex translation model that handles both forward and reverse directions. By combining separate models into one integrated architecture with shared components and directional pathways, the system reduces overall complexity while maintaining full translation capability in both directions.
Solution Approach 2:
The unified model achieves multi-functionality by incorporating directional pathways and parameter sharing mechanisms that enable it to perform multiple translation directions simultaneously. This universal architecture reduces system complexity compared to deploying separate specialized models for each direction.
3Adaptability or versatility
If multi-task learning techniques are used to train unified models, then duplex translation is enabled, but performance and efficiency deteriorate
Solution Approach 1:
The model segments the multi-task learning problem into distinct directional pathways with task-specific components for each translation direction. This segmentation prevents task interference while maintaining the ability to perform multiple translation directions, thereby preserving performance and efficiency that would otherwise deteriorate in a purely shared multi-task architecture.
Solution Approach 2:
The architecture applies local quality by having direction-specific components (encoders, decoders, attention mechanisms) tailored to each translation pathway while sharing lower-level features. This allows each directional task to have optimized local structures for its specific language pair, preventing performance deterioration that would occur with uniform multi-task sharing.
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques that may use machine learning for reversible translations of speech utterances. The techniques include training and using duplex neural networks (NNs) having a first subnetwork and a second subnetwork that are mirror images of each other. Training data for training the duplex NNs may include a target output that includes a first speech utterance in a first language, a first training input that includes the target output distorted by a noise, and a second training input that includes a second speech utterance in a second language. The duplex NNs may be trained to identify, using the first training input and the second training input, at least one of the target output or the first noise.