Distilled Sentence Embedding Model for Efficient Similarity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language models, particularly those using cross-attention (CA) operations, face computational inefficiencies in sentence similarity scoring due to time-consuming inference phases and the lack of training for producing sentence embeddings relevant to the task at hand.
Innovation Solution
The development of a distilled sentence embedding (DSE) language model through neural network distillation, which decouples sentence embeddings from similarity function analysis, allowing for precomputation of candidate sentence representations and efficient similarity scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If cross-attention operations are applied in a cascade throughout a stack of multi-head attention layers to compute similarity between sentence pairs, then the accuracy of sentence similarity comparisons is improved, but the processing time and computational cost increase excessively
Solution Approach 1:
The patent segments the transformer language model into two separate components: a sentence embedding model that generates sentence representations, and a similarity function that computes similarity scores. This segmentation allows the embedding model to be trained independently to produce task-specific sentence embeddings, while the similarity function handles the comparison task, thereby reducing the computational burden during inference while maintaining accuracy.
Solution Approach 2:
The patent applies preliminary action by training the sentence embedding model in advance using knowledge distillation from the teacher transformer model. This pre-computation of sentence embeddings allows the system to store and reuse embeddings for multiple sentences, avoiding the need to repeatedly run the full cross-attention operations during similarity scoring, thus significantly reducing processing time.
2Measurement precision
If cross-attention operations are applied in a cascade throughout a stack of multi-head attention layers to compute similarity between sentence pairs, then the accuracy of sentence similarity comparisons is improved, but the computational cost increases excessively
Solution Approach 1:
The patent segments the transformer language model into two separate components: a sentence embedding model that generates sentence representations, and a similarity function that computes similarity scores. This segmentation allows the embedding model to be trained independently to produce task-specific sentence embeddings, while the similarity function handles the comparison task, thereby reducing the computational burden during inference while maintaining accuracy.
Solution Approach 2:
The patent changes the parameters by using knowledge distillation to transfer knowledge from the large teacher model to a smaller student embedding model. This results in a more compact model with fewer parameters that can generate high-quality sentence embeddings with significantly lower computational cost during inference.
3Productivity
If language models are trained to produce sentence embeddings with respect to the task at hand, then the efficiency of sentence similarity scoring is improved, but the complexity of the training process increases
Solution Approach 1:
The patent uses knowledge distillation as an intermediary process to transfer knowledge from the teacher transformer model to the student embedding model. This intermediary training approach simplifies the training process by using the pre-trained teacher model as a guide, allowing the student model to learn effective sentence embeddings without requiring complex multi-stage training procedures.
Solution Approach 2:
The patent employs copying by using the sentence embeddings generated by the teacher model as target labels during the training of the student embedding model. This copying approach allows the student model to learn by imitating the teacher's embeddings, simplifying the training process while maintaining high efficiency in sentence similarity scoring.
Data Source
AI summary
The disclosure herein describes a system and method for attentive sentence similarity scoring. A distilled sentence embedding (DSE) language model is trained by decoupling a transformer language model using knowledge distillation. The trained DSE language model calculates sentence embeddings for a plurality of candidate sentences for sentence similarity comparisons. An embedding component associated with the trained DSE language model generates a plurality of candidate sentence representations representing each candidate sentence in the plurality of candidate sentences which are stored for use in analyzing input sentences associated with queries or searches. A representation is created for the selected sentence. This selected sentence representation is used with the plurality of candidate sentence representations to create a similarity score for each candidate sentence-selected sentence pair. A retrieval component identifies a set of similar sentences from the plurality of candidate sentences responsive to the input query based on the set of similarity scores.