Document Search Engine Shingle Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional document search engines fail to effectively group documents with similar content, leading to cluttered and repetitive search results, as existing filters do not adequately distinguish between documents with minor edits or similar content.
Innovation Solution
A document search engine builds and indexes 'shingles' – contiguous sequences of text items – to identify and group documents by content similarity, using these shingles to filter and curate search results, thereby improving relevance and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional filters are used to narrow search results, then the quantity of search results is reduced, but documents with similar content (e.g., different versions with minor edits) are not adequately distinguished and remain in the results
Solution Approach 1:
The patent segments documents into overlapping contiguous sequences of text items called 'shingles'. Each document is divided into multiple shingles that represent different portions of the content. This segmentation enables precise comparison of document contents by comparing their shingles rather than treating documents as monolithic units, thereby improving the ability to distinguish between documents with similar content.
Solution Approach 2:
The patent changes the parameter of document representation from whole-document comparison to shingle-based comparison. By transforming documents into sets of shingles and comparing these sets, the system achieves more accurate content similarity detection. The shingle approach allows the system to capture local content patterns and distinguish between documents that differ even slightly in content.
2Loss of information
If all search results are presented to users, then completeness of information is maintained, but user analysis time increases due to cluttered and repetitive results
Solution Approach 1:
The patent performs preliminary grouping of documents by content similarity before presenting search results to users. By pre-computing shingles for all documents and organizing them in a shingles data store, the system can quickly identify and group similar documents when processing search queries. This preliminary organization eliminates the need for users to manually analyze cluttered results, as similar documents are already grouped together in the search results.
3Measurement precision
If detailed content analysis is performed on all documents to improve search accuracy, then measurement precision is improved, but processing resources are overwhelmed
Solution Approach 1:
The patent segments the content analysis task into comparing shingles rather than analyzing entire documents. Each shingle is a small contiguous sequence of text items, making comparison computationally efficient. The shingles data store organizes shingles by their content, enabling quick lookup and comparison without requiring full document analysis, thereby maintaining processing efficiency while achieving detailed content comparison.
Solution Approach 2:
The patent performs preliminary processing to build shingles for all documents during indexing and stores them in an optimized shingles data store. This preliminary action transforms the complex task of detailed content analysis into a simpler task of comparing pre-computed shingles. When search queries are processed, the system leverages this pre-computed structure to efficiently identify similar documents without performing exhaustive content analysis at query time.
Data Source
AI summary
Implementations of the disclosure describe grouping documents by content similarity. A method of the disclosure includes determining a first set of document identifiers (IDs) for documents identified responsive to a search query, for each document ID of the first set: obtaining shingles corresponding to the document ID, identifying a second set of document IDs corresponding to the obtained shingles, grouping the identified document IDs of the second set in a grouping data structure, and determining resulting document IDs from the grouping data structure, the resulting document IDs corresponding to a threshold number of the document IDs in the grouping data structure having a highest number of occurrences. The method further includes associating the documents corresponding to the resulting documents IDs with one another in search results generated responsive to the search query.