Fused Multi-Encoder Tokenization for Redundancy-Reduced Representations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks struggle to efficiently process multiple encoding sequences from different domains and modalities, leading to redundant representations that result in larger than necessary encoded data, which is inefficient for storage and processing.
Innovation Solution
A system that combines multiple encoding sequences using a deduplicator neural network to remove redundancies and generate a single, efficient, discrete representation, utilizing a tokenizer to identify and encode the deduplicated sequences using a codebook, and a reduplicator neural network to reconstruct the original data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple encoding sequences from different domains and modalities are processed separately by specialized encoders, then the representation accuracy for each domain is improved, but the overall data redundancy increases and storage efficiency deteriorates
Solution Approach 1:
The patent combines multiple encoding sequences from different domains and modalities into a single unified encoding sequence. The system integrates representations from specialized encoders (e.g., audio encoder, video encoder, text encoder) that process different aspects of the same data, merging them into one compact sequence that preserves domain-specific information while eliminating redundancy across modalities.
Solution Approach 2:
The patent creates a universal encoding sequence that serves multiple functions simultaneously. This single encoding sequence can represent different domains (audio, video, text) and modalities (spoken language, written language, visual features) that were previously handled by separate specialized encoders, making the encoding system multi-functional and domain-agnostic.
2Loss of information
If multiple specialized encoders are used for different domains and modalities, then the information completeness is improved, but the encoding efficiency and storage compactness deteriorate
Solution Approach 1:
The system merges multiple encoding sequences generated by specialized encoders into a single unified sequence. This consolidation maintains information completeness by preserving the essential characteristics from all domain-specific encoders while achieving encoding efficiency through the elimination of redundant information across different modalities and domains.
Data Source
AI summary
Methods and systems for one or more computers, in which a method includes obtaining encoding sequences of an input data item, in which each encoding sequence includes a respective encoding vector at each position of multiple positions. The method includes generating a combined encoding sequence by, at each position, combining the respective encoding vectors at the position in the multiple of encoding sequences. The method includes processing the combined encoding sequence using a deduplicator neural network to generate a deduplicated encoding sequence that includes a respective deduplicated encoding vector for each of the positions and applying a tokenizer to the deduplicated encoding sequence to identify, for each deduplicated encoding vector, a discrete representation of the deduplicated encoding vector generated from respective codebook vectors from each of a set of one or more codebooks, in which each codebook is a respective discrete set of codebook vectors.