Machine-Trained Term Frequency Weighting for Dense Embedding Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current text-processing engines face challenges in generating embedding vectors that effectively capture semantic information and distinguish between related but distinct text instances, leading to poor retrieval performance and difficulties in combining high-dimensional sparse vectors from different encoders.
Innovation Solution
A computer-implemented technique generates an embedding vector by transforming a high-dimensional term-frequency vector using machine-trained neural networks, applying diagonal and full matrices of weighting factors to produce a dense, compact representation that captures semantic information and relationships between terms, allowing for improved retrieval and combination with other dense vectors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional encoding methods are used to generate embedding vectors, then the process is simple, but the retrieval performance is poor and semantic information is not effectively captured
Solution Approach 1:
The patent transforms the embedding vector representation parameters by converting from high-dimensional sparse vectors to low-dimensional dense vectors. This parameter change enables better retrieval performance while reducing dimensionality, resolving the contradiction between reliability and complexity through mathematical transformation rather than process simplification
Solution Approach 2:
The patent replaces traditional mechanical encoding processes with machine-learning-based neural network models. These models automatically learn semantic relationships and generate high-quality embedding vectors, improving retrieval performance while the automated nature of ML reduces manual complexity
2Adaptability or versatility
If high-dimensional sparse vectors are used to represent text, then term frequency information is preserved, but the vectors cannot be readily combined with other dense vectors from different encoders
Solution Approach 1:
The patent changes the parameter structure of embedding vectors from sparse high-dimensional format to dense low-dimensional format. This transformation enables seamless combination with other dense vectors from different encoders while maintaining information density through learned semantic representations
Solution Approach 2:
The patent creates a composite representation by integrating term frequency information with learned semantic features in a unified dense vector format. This composite structure combines the advantages of both sparse and dense representations, enabling versatility in combining with other models while preserving information
3Reliability
If all terms in input text are treated equally in encoding, then the process is straightforward, but the ability to de-emphasize certain terms for better retrieval is lost
Solution Approach 1:
The patent applies local quality by assigning different weighting factors to different terms in the input text. Important terms receive higher weights while less important terms receive lower weights, enabling selective emphasis that improves retrieval accuracy without requiring complex manual intervention
Solution Approach 2:
The patent replaces manual term weighting with automated machine-learning-based weighting mechanisms. The neural network models automatically learn and apply appropriate weights to different terms based on their semantic importance, improving retrieval accuracy while the automation reduces operational complexity
Data Source
AI summary
A computer-implemented technique generates a dense embedding vector that provides a distributed representation of input text. The technique includes: generating an input term-frequency (TF) vector of dimension g that includes frequency information relating to frequency of occurrence of terms in an instance of input text; using a TF-modifying component to modify the term-specific frequency information in the input TF vector by respective machine-trained weighting factors, to produce an intermediate vector of dimension g; using a projection component to project the intermediate vector of dimension g into an embedding vector of dimension k, where k is less than g. Both the TF-modifying component and the projection component use respective machine-trained neural networks. An application performs any of a retrieval-based function, a recognition-based function, a recommendation-based function, a classification-based function, etc. based on the embedding vector.