Learned Term Weighting for Document Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector-based similarity measures in natural language processing and information retrieval lack effectiveness in incorporating additional information such as term position and anchor text, leading to suboptimal performance in tasks like query suggestion and ad filtering.
Innovation Solution
A learning framework that represents objects as vectors and learns the weights of these vectors using labeled training data to minimize a loss function, allowing for the incorporation of various features like term frequency, document frequency, query frequency, and HTML-specific features to compute improved similarity scores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional TF-IDF schemes are used to construct term vectors, then the method is simple and computationally efficient, but the similarity measure quality is suboptimal because additional information such as term position and anchor text cannot be easily incorporated
Solution Approach 1:
The patent changes the parameters of term vector construction by introducing learned weight functions that incorporate multiple features (term frequency, document frequency, query frequency, HTML features) instead of using fixed TF-IDF weights. This allows the system to capture additional information while maintaining a vector-based framework.
Solution Approach 2:
The patent makes the term weighting dynamic by learning weights from training data using a loss function. The weights are not fixed but are adapted based on the specific task and data, allowing the system to optimize for different applications such as query suggestion or ad relevance.
2Ease of manufacture
If all words in a document are included in the term vector using bag-of-words strategy, then the construction is simple, but the vector dimensionality becomes very high leading to computational inefficiency
Solution Approach 1:
The patent extracts only the most relevant terms for the specific task by using learned weight functions that can identify and prioritize important terms. This reduces the effective dimensionality of the term vectors by focusing on task-relevant features rather than including all words.
3Ease of operation
If fixed weight schemes are used for term vectors, then the method is easy to implement, but the system cannot adapt to different tasks such as query suggestion or ad relevance
Solution Approach 1:
The patent implements dynamic weight learning that adapts to different tasks. The same framework can be applied to query suggestion, ad relevance, or other tasks by training on appropriate data, making the system versatile while maintaining ease of implementation through a unified approach.
Solution Approach 2:
The patent creates a universal term weighting framework that can serve multiple functions. The learned weight functions and feature extraction mechanisms work across different applications, allowing the same system to improve various NLP tasks without requiring task-specific customization.
Data Source
AI summary
Described is a technology for measuring the similarity between two objects (e.g., documents), via a framework that learns the term-weighting function from training data, e.g., labeled pairs of objects, to develop a learned model. A learning procedure tunes the model parameters by minimizing a defined loss function of the similarity score. Also described is using the learning procedure and learned model to detect near duplicate documents.