Partitioned Lexicographic Search System for Large Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data stores face inefficiencies in lexicographic similarity-based searches due to the need to calculate similarity metrics for every entry, which becomes intractable as the number of entries grows, leading to prolonged search times and increased processing resources.
Innovation Solution
A partitioned lexicographic similarity-based search system is implemented, where index entries are distributed across partitions based on similarity metrics, allowing for intelligent pruning of search spaces by selecting only the most similar partitions, using genetic algorithms to optimize collector distribution and reduce the number of similarity calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If similarity metrics are calculated for every entry in large data stores, then search completeness is maintained, but search time and processing resources become intractable
Solution Approach 1:
The patent divides the large data store into multiple partitions, each containing a subset of index entries. Instead of calculating similarity metrics for all entries, the system calculates similarity between the search term and partition representatives (collectors) to identify relevant partitions, then performs detailed similarity calculations only within those selected partitions. This segmentation reduces the search space from the entire data store to a manageable subset while maintaining search completeness.
2Measurement precision
If similarity metrics are calculated for every entry in large data stores, then accurate near-matches are found, but processing resources increase excessively
Solution Approach 1:
The system performs preliminary actions by pre-calculating and storing partition representatives (collectors) that characterize each partition's content. Before performing similarity calculations, the system identifies relevant partitions by comparing the search term against these pre-computed collectors. This preliminary filtering step avoids the need to calculate similarity metrics for every entry, significantly reducing processing resources while maintaining the ability to find accurate near-matches in the selected partitions.
3Loss of information
If all index entries are searched, then comprehensive search results are obtained, but the complexity of the search system increases
Solution Approach 1:
The patent introduces partition representatives (collectors) as intermediary elements between the search term and the actual index entries. These collectors serve as mediators by providing a condensed representation of each partition's content. The system first compares the search term against collectors to identify relevant partitions, then performs detailed searches only within those partitions. This intermediary layer simplifies the overall search system complexity by avoiding the need to directly compare the search term against every single index entry while maintaining comprehensive search results.
Data Source
AI summary
A search method is provided. The method comprises, building a first plurality of collectors, associating a first plurality of index entries with the first plurality of collectors, wherein each index entry comprises a search target and is associated with at least one of the first plurality of collectors based on a similarity between the search target of the index entry and the at least one collector, selecting a second plurality of collectors from among the first collectors based on a similarity between a search term and each of the second plurality of collectors, selecting a second plurality of index entries from among the index entries associated with the second plurality of collectors based on a similarity between the search term and the search target of each of the index entries associated with the second plurality of collectors, and presenting information related to the second plurality of index entries.