Conditional Masked Language Modeling for Unsupervised Sentence Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State-of-the-art language models struggle to produce effective language segment embeddings without supervised or semi-structured data, making it difficult to cover diverse domains and languages, and internal masked language model representations are not directly suitable for generating good language segment embeddings.
Innovation Solution
The proposed method, Conditional Masked Language Modeling (CMLM), trains machine learning models to generate contextual language embeddings by processing language segments with a first encoding model, generating masked versions, and modifying parameters based on predictions compared to masked tokens, allowing for unsupervised learning of language segment representations on large-scale unlabeled corpora.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised tasks or semi-structured data are used for training language models, then language segment embedding performance is improved, but data acquisition cost and complexity increase
Solution Approach 1:
The model performs masked language modeling where it predicts masked tokens using contextual information from the same sentence, making the model self-sufficient and eliminating the need for external labeled data or complex semi-structured annotations
Solution Approach 2:
The contextual language embedding serves as an intermediary representation that bridges the input sentence and the masked token prediction task, enabling the model to learn effective embeddings through an unsupervised preprocessing step
2Productivity
If masked language model pre-training is performed on large scale unlabeled corpora, then data coverage and efficiency are improved, but direct language segment embedding quality deteriorates
Solution Approach 1:
The training process is segmented into two distinct stages: first generating contextual language embeddings through forward propagation, then using these embeddings to condition masked token predictions, allowing each stage to optimize for its specific goal
Solution Approach 2:
The approach adds a new dimension to the training process by conditioning the masked language model on external contextual embeddings, transforming the standard MLM objective into a conditional task that produces better sentence representations
Data Source
AI summary
The present disclosure provides a novel sentence-level representation learning method Conditional Masked Language Modeling (CMLM) for training on large scale unlabeled corpora. CMLM outperforms the previous state-of-the-art English sentence embedding models, including those trained with (semi-)supervised signals. For multilingual representations learning, it is shown that co-training CMLM with bitext retrieval and cross-lingual natural language inference (NL) fine-tuning achieves state-of-the-art performance. It is also shown that multilingual representations have the same language bias and principal component removal (PCR) can eliminate the bias by separating language identity information from semantics.