Document Similarity Search Using Least-Frequent Terms and Proximity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document search techniques, such as file hash and search term matching, exhibit poor performance in finding similar content within organizations, particularly in security investigations, regulatory compliance, or litigation scenarios.
Innovation Solution
A search system that analyzes sample documents for term frequency and global frequency, applies proximity-weighted scoring, and generates bespoke queries to identify and rank terms based on their significance and proximity, reducing computational costs by using keyword searches instead of vector-based similarity searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional file hash or search term matching techniques are used, then the search process is simple to implement, but the performance in finding similar content is poor
Solution Approach 1:
The patent transforms the search approach by changing parameters from exact matching (file hash) to similarity scoring based on term frequency, inverse document frequency, and proximity weights. This allows the system to find semantically similar documents rather than only exact matches, significantly improving reliability while managing complexity through structured computational methods
Solution Approach 2:
The system performs preliminary actions by pre-calculating term frequencies, inverse document frequencies, and proximity weights for all documents before the actual search query. This pre-processing creates indexes that enable fast similarity calculations during search operations, improving accuracy without making the search process itself overly complex
2Reliability
If comprehensive document analysis with multiple scoring factors is performed, then the search accuracy improves, but the computational cost increases
Solution Approach 1:
The patent applies partial action by focusing computational effort on the most discriminating features: term frequency, inverse document frequency, and proximity of key terms. Rather than analyzing every aspect of document similarity equally, the system weights certain factors more heavily, achieving good accuracy with reduced computational overhead compared to comprehensive analysis
Solution Approach 2:
The search process is segmented into distinct computational stages: term extraction, frequency calculation, proximity analysis, and similarity scoring. This segmentation allows the system to process documents in manageable steps, optimizing resource usage at each stage while maintaining overall search accuracy
3Productivity
If exact search term matching is used, then the search operation is fast, but the ability to find semantically similar documents is limited
Solution Approach 1:
The patent introduces dynamics by replacing static exact matching with dynamic similarity scoring. The system adapts the search results based on multiple factors including term frequency variations, document-specific term importance (inverse document frequency), and contextual proximity. This dynamic approach maintains reasonable search speed while dramatically improving flexibility in finding semantically similar documents
Data Source
AI summary
A computer-implemented method for performing a document search action includes receiving a search request identifying a sample document. A plurality of terms occurring within the sample document is identified. Each term is scored based on a global term frequency of that term within a collection of documents. Each term is then weight based on proximity, within the sample document, of that term to at least one of the other terms of the first plurality of terms. A document search query is created to include a proximity-limiting clause restricting results to include documents that have a highest-weighted term of the first plurality of terms within a threshold distance of another term of the first plurality of terms. The document search query is performed, thereby resulting in identification of a first resulting document.