Frozen Index Search via Throttled Thread Pool and Shard Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data communication networks in distributed application structures face challenges in efficiently managing and searching large indices that exceed hardware limits, leading to increased memory usage and slower search response times, especially when dealing with frozen indices which require significant resources but are rarely accessed.
Innovation Solution
Implementing a throttled thread pool with a First-In-First-Out (FIFO) queue to manage search requests for frozen indices, where shards are loaded into memory only when queried, and using a pre-check phase to determine if documents are within the search time frame, reducing memory consumption and improving search efficiency by limiting concurrent searches and optimizing resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If frozen indices are kept in memory for fast searching, then search speed is improved, but memory usage increases significantly
Solution Approach 1:
The system dynamically loads index shards into memory only when they are needed for searching, and removes them when not needed. This transforms the static memory allocation approach into a dynamic one, allowing the system to adapt memory usage based on actual search demands rather than keeping all frozen indices permanently in memory.
Solution Approach 2:
The frozen index is divided into multiple shards that can be independently managed. Each shard can be loaded into memory separately when needed, rather than loading the entire index at once. This segmentation allows selective loading of only the necessary portions of the index, reducing overall memory consumption.
2Productivity
If multiple searches are performed concurrently on frozen indices, then search throughput is improved, but resource consumption increases
Solution Approach 1:
The system uses a throttled thread pool that controls the rate and timing of search operations on frozen indices. By regulating when searches occur and limiting concurrent search threads, the system performs searches in a controlled, periodic manner rather than allowing unbounded concurrent execution, thus managing resource consumption while maintaining productivity.
3Reliability
If all index shards are loaded into memory, then search completeness is improved, but memory efficiency deteriorates
Solution Approach 1:
The system performs a pre-check phase before actual searching to determine which shards contain documents within the search time frame. This preliminary action identifies only the necessary shards that need to be loaded into memory, ensuring search completeness for relevant data while avoiding loading unnecessary shards that would waste memory.
Solution Approach 2:
The pre-check mechanism extracts and identifies only the specific shards that are relevant to the current search query based on time frame criteria. Instead of loading all shards, the system extracts and loads only the necessary subset, maintaining search completeness for relevant documents while improving memory efficiency.
Data Source
AI summary
Methods and systems for searching a frozen index are provided. Exemplary methods include: a method may comprise: receiving an initial search and a subsequent search; loading the initial search and the subsequent search into a throttled thread pool, the throttled thread pool including; getting the initial search from the throttled thread pool; storing a first shard from a mass storage in a memory in response to the initial search; performing the initial search on the first shard; providing first top search result scores from the initial search; and removing the first shard from the memory when the initial search is completed.