Search Index Sorting for Selective Result Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search systems face inefficiencies in processing search queries over large data sets, as they require searching the entire index and pulling large intermediate result sets into memory, leading to high latency and computational costs.
Innovation Solution
A system and method that selectively retrieve search results by identifying index portions matching filter criteria, sorting them in a predefined order, and sequentially retrieving identifiers of matching entities, thereby reducing the need to pull large result sets into memory and allowing for controlled latency through adjustable search result limits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire index is searched for index entries matching the search query, then all matching results are obtained, but search time scales with the size of the overall data set leading to high latency
Solution Approach 1:
The index is divided into multiple index portions, each sorted in a predefined sort order. The system identifies and searches only the relevant index portions that contain matching entries, rather than scanning the entire index. This segmentation allows the search to focus on smaller, targeted subsets of data, reducing search time while maintaining result completeness.
Solution Approach 2:
Index portions are pre-sorted in a predefined sort order before search operations. This preliminary organization allows the system to efficiently locate and retrieve matching entries without performing sorting during the search operation, significantly reducing search latency while ensuring results are returned in the correct order.
2Reliability
If large intermediate result sets are pulled into memory for comparison, then all matching entities can be identified, but memory usage and processing time increase dramatically
Solution Approach 1:
The system extracts and processes only the necessary index portions and their associated identifiers rather than loading entire intermediate result sets into memory. By pulling out only the relevant sorted index portions and comparing identifiers directly, the system maintains accurate matching entity identification while dramatically reducing memory consumption and processing overhead.
Solution Approach 2:
Instead of working with large intermediate result sets, the system uses identifier representations from the pre-sorted index portions. These identifiers serve as lightweight copies or references that enable efficient comparison and matching without requiring the full data structures to be loaded into memory, improving processing efficiency while maintaining accuracy.
3Reliability
If the search retrieves all matching results regardless of quantity, then complete search coverage is achieved, but latency cannot be controlled and may be excessively high
Solution Approach 1:
The system dynamically adjusts the search process based on the predefined sort order and identifier comparison. By organizing index portions in advance and using efficient identifier matching, the system can control the number of comparisons needed, making search latency predictable and controllable while maintaining complete result coverage through the systematic traversal of sorted index portions.
Data Source
AI summary
A search server system having one or more processors and memory receives a search query including filter criteria. The filter criteria specify operands and a logical relationship. In response to the search query the server identifies index portions sorted in a same sort order. The server identifies matching entities that match the search query by identifying index entries having a first identifier and matching filter(s) of a first operand and identifying, index entries having a second identifier that is sequentially adjacent to the first identifier in the sort order and matching filter(s) of a second operand. When index portion(s) associated with the first operand include index entries having a respective identifier between the first identifier and the second identifier and matching the filter(s) of the first operand, the server marks a respective entity associated with the respective identifier as matching the search query and sends search results to the client.