Multilingual Vision-Language Embedding Without Machine Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vision-language systems suffer from inflexibility, inaccuracy, and inefficiency, particularly when handling multiple languages, due to inadequate training data, misaligned text encoders, and the need for machine translation, which consumes computational resources and introduces errors.
Innovation Solution
A multilingual vision language system that trains a multilingual large language model to embed text into the embedding space of a vision language model using contrastive loss and cross-lingual teacher learning, without adjusting the vision encoder, and employs translation-resampling to balance training data across languages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine translation is used to handle multilingual queries, then the system can process multiple languages, but computational resources are consumed and errors are introduced
Solution Approach 1:
The system pre-trains the text encoder with multilingual data before deployment, so that when queries arrive, the encoder is already capable of handling multiple languages without requiring real-time translation. This preliminary preparation eliminates the need for computational translation during query processing.
Solution Approach 2:
The patent introduces an embedding space as an intermediary representation that bridges different languages. By mapping text from various languages into a unified embedding space that aligns with image representations, the system avoids direct machine translation and instead uses semantic alignment for cross-lingual understanding.
2Adaptability or versatility
If machine translation is used to handle multilingual queries, then the system can process multiple languages, but translation errors are introduced
Solution Approach 1:
The unified embedding space serves as a mediator that preserves semantic information across languages without requiring translation. By directly mapping multilingual text and images into the same vector space, the system maintains accuracy while achieving multilingual capability, avoiding the error-prone translation step entirely.
Solution Approach 2:
The system changes the parameters of the text encoder through contrastive training on multilingual data, transforming it from a monolingual to a multilingual model. This parameter adjustment enables the encoder to directly process multiple languages with high accuracy, eliminating translation errors.
3Adaptability or versatility
If the vision encoder is adjusted during training, then the model can adapt to new data, but training time and computational resources increase
Solution Approach 1:
The patent segments the training process into two distinct phases: first, the vision encoder is pre-trained on image data; second, the text encoder is trained separately with multilingual data while the vision encoder remains frozen. This segmentation allows the text encoder to adapt to multilingual data without requiring retraining of the entire model, thus reducing training time.
Solution Approach 2:
The system dynamically adjusts which components are trainable at different stages: initially both encoders are trained, then the vision encoder is frozen while only the text encoder continues training. This dynamic approach allows adaptability where needed while conserving computational resources where not needed.
4Ease of manufacture
If a single-language text encoder is used, then the system is simple to implement, but it cannot handle multiple languages
Solution Approach 1:
The patent transforms a single-language text encoder into a universal multilingual encoder by training it on diverse multilingual data. The same encoder architecture and parameters serve multiple languages, achieving multi-functionality without complicating the system implementation. This universal encoder can process text in any of the trained languages using the same computational path.
Data Source
AI summary
The present disclosure relates to systems, non-transitory computer-readable media, and methods for training a multilingual large language model to embed text into an embedding space of a vision language model comprising a text encoder for a first language and a vision encoder. In particular, in some embodiments, the disclosed systems generate, utilizing the vision encoder, image embeddings for images. Additionally, in some embodiments, the disclosed systems generate, utilizing the multilingual large language model, text embeddings for text in languages other than the first language. Furthermore, in some embodiments, the disclosed systems determine similarity metrics between the image embeddings for the images and the text embeddings for the text. Moreover, in some embodiments, the disclosed systems adjust parameters of the multilingual large language model to reduce an output of a contrastive loss function based on the similarity metrics without adjusting parameters of the vision encoder.