Document Matching Engine Using Signature-Based Relevance Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engine technologies face inefficiencies in accurately measuring document relevance, leading to low precision rates and a high return of irrelevant documents, particularly due to language dependency and resource-intensive models.
Innovation Solution
A relevance detection engine that generates signatures for documents based on tokenization, ranking, and normalization, allowing for efficient calculation of similarity between documents and filtering out irrelevant results by a predetermined percentage threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional search engine models (term vector-space model, probabilistic model, latent semantic space model) are used to measure document relevance, then comprehensive document analysis is performed, but the computing resources required are highly intensive and measurement accuracy is often inaccurate
Solution Approach 1:
The patent segments the document analysis process into distinct phases: signature generation (extracting key terms and creating compact representations) and relevance detection (comparing signatures using efficient algorithms). This segmentation allows the system to avoid computationally intensive full-document analysis while maintaining measurement accuracy through carefully selected signature elements.
Solution Approach 2:
The patent creates simplified copies of documents in the form of signatures that capture essential relevance information. These signatures serve as lightweight proxies for full documents, enabling rapid comparison and relevance assessment without processing the entire document content, thus reducing computing resource consumption significantly.
2Measurement precision
If conventional search engines return a large list of documents to ensure coverage, then more potential relevant documents are included, but the precision rate of retrieval is low due to many irrelevant documents
Solution Approach 1:
The patent performs preliminary filtering by comparing query signatures against document signatures before returning full documents. The relevance detection engine uses the signatures to pre-assess document relevance, filtering out obviously irrelevant documents early in the process. This preliminary action reduces the number of documents that need to be returned while maintaining high precision, as only documents with sufficiently similar signatures are included in the results.
3Measurement precision
If conventional search engines implement different language parsers and analyzers for each written language, then language-specific accuracy is improved, but resource usage increases significantly
Solution Approach 1:
The patent implements a universal signature generation approach that can process multiple languages through a single unified system. The signature generator extracts key terms and creates representations that capture semantic meaning across different languages without requiring separate complex language-specific analyzers. This universal approach maintains language-specific accuracy while significantly reducing resource consumption by eliminating the need for multiple specialized processing pipelines.
Data Source
AI summary
A system generates an output of documents having with a particular relevance range. The system receives an initial document comprising text, a list of documents for matching, each document comprising text, and a minimum substring match length. The system normalizes the text of the documents of the list of documents. The system searches common sub-strings between the text of the initial document and the text of each document of the list of documents. The system calculates a match percentage based on the search common sub-strings and outputs documents having a match percentage corresponding to a predetermined value. Also disclosed is a process for generating an output of documents within a particular relevance range.