Search Index Document Ranking Using Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The exponential growth of web documents poses a challenge for search engines, as only a segment is indexed, leaving many documents in the 'Deep Web' unprocessed, and processing all documents is computationally expensive, with varying usefulness, leading to inefficient resource allocation and user satisfaction issues.
Innovation Solution
A method and system using a machine learning algorithm to selectively index documents based on user interactions and size, prioritizing documents with higher usefulness and smaller size, maximizing the cumulative quality value parameter within storage limits, employing algorithms like LambdaMART for ranking and selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all web documents are processed and indexed, then the completeness of the search index is improved, but the computational cost and resource consumption increase exponentially
Solution Approach 1:
The patent changes the parameter of document selection from exhaustive processing to selective processing based on predicted usefulness scores. Documents are ranked by a machine learning model that predicts their utility, and only top-ranked documents are indexed, transforming the approach from complete to selective based on predicted value parameters
Solution Approach 2:
Instead of processing all documents (excessive action), the system processes only a partial subset of documents that are predicted to be most useful. This partial action achieves sufficient search quality while dramatically reducing computational costs and resource consumption
2Quantity of substance
If more documents are included in the search index, then the coverage of search results is improved, but the storage requirements and bandwidth consumption increase
Solution Approach 1:
The system changes from indexing all documents to indexing a selective subset based on predicted usefulness. By ranking documents and selecting only those above a threshold, the patent reduces the quantity of indexed documents while maintaining search effectiveness, thereby reducing storage requirements and bandwidth consumption
3Loss of information
If a larger portion of web documents is indexed, then the depth of information retrieval is improved, but the processing time and computational resources increase
Solution Approach 1:
The patent applies preliminary action by predicting the usefulness of documents before indexing them. The machine learning model pre-ranks documents based on their expected utility, allowing the system to selectively index only high-value documents in advance, thereby reducing processing time while maintaining information retrieval depth
Solution Approach 2:
The system changes the processing approach from uniform processing of all documents to differential processing based on predicted usefulness scores. High-scoring documents are processed and indexed, while low-scoring documents are skipped, reducing overall processing time while preserving retrieval depth for important information
Data Source
AI summary
A method and system for selecting documents for inclusion into a search engine search index executed by a machine learning algorithm (MLA) comprising: acquiring a plurality of digital documents, each respective document having a respective size, determining a respective value parameter based on user interactions, ranking the digital documents to obtain a ranked list of documents based on: a respective quality value parameter based on the respective value parameter and the respective size, the ranking being performed such that a cumulative quality value parameter is maximized based on: the quality value parameter of a given digital document is weighted by a sum of the sizes of a given subset of documents, wherein the given subset consists of documents having a respective value parameter below the value parameter of the given digital document, selecting a subset from the ranked list and storing the subset in the search index.