Category-Based Search Ranking Using Local Document Frequency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional information retrieval systems face improper ranking of search results when using global statistics for category-based searches, leading to relevant documents being ranked low or discarded, due to the impracticality of maintaining separate indices for each category.
Innovation Solution
The method calculates and stores histograms for index terms and categories, allowing for the estimation of local document frequency within a specified category restriction, using a single comprehensive index to rank documents as if only local statistics were considered.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate indices are maintained for each category to achieve accurate local statistics, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple category-specific indices into a single comprehensive index structure. The index contains documents from all categories with their category identifiers stored alongside the documents. This consolidation reduces the complexity of maintaining and searching multiple separate indices while preserving the ability to perform category-based searches through filtering on the unified index.
Solution Approach 2:
The patent introduces a category identifier field as an intermediary element within the unified index. This intermediary allows the system to distinguish between different categories without requiring separate indices, enabling efficient category-based filtering and statistics calculation on the consolidated index structure.
2Device complexity
If global statistics are used for category-based searches, then device complexity is reduced, but measurement precision deteriorates
Solution Approach 1:
The patent applies local quality by calculating and applying category-specific document frequency statistics for each category rather than using global statistics across all categories. This allows the search algorithm to use locally accurate term frequencies and document frequencies within each category, improving the precision of relevance scoring and ranking for category-constrained searches.
3Measurement precision
If multiple indices are maintained for each category, then measurement precision is improved, but loss of substance increases
Solution Approach 1:
The patent combines multiple category-specific indices into a single unified index structure that stores all documents with their category identifiers. This merging eliminates the need to maintain duplicate index structures for each category, significantly reducing memory consumption while preserving the ability to calculate accurate local document frequencies through category-based filtering on the unified index.
Data Source
AI summary
A method for searching a document collection includes providing an index of terms indicating the documents in which the terms appear. A first statistical distribution of each of at least some of the terms in the index and a second statistical distribution of each of at least some of the categories are estimated a over the documents in the collection. A query including one or more of the terms and a category restriction referring to at least one of the categories is accepted. A modified term distribution is produced by operating on the first statistical distribution of at least one of the terms in the query using the second statistical distribution, responsively to the category restriction. The query is applied to the index to return a response, in which occurrences of the at least one of the terms are scored responsively to the modified term distribution.