Search Node Index File Warming via Dynamic Query Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search systems face inefficiencies when transitioning to new index files, requiring service interruption and the use of pre-prepared warm-up queries to load pages into cache, which can cause latency and disrupt ongoing searches.
Innovation Solution
Determine warm-up queries from actual queries received during service, using a gradually-increased query replication rate to concurrently warm up new index files without interrupting service, allowing continuous operation and reducing I/O impact on disks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-prepared warm-up queries are used to load pages into cache before switching to new index files, then cache warming up is achieved, but service interruption and search latency occur
Solution Approach 1:
The system performs preliminary actions by generating and executing warm-up queries before the index file switch is complete. These warm-up queries proactively load necessary pages into cache memory in advance, ensuring that when the switch occurs, the cache is already populated with frequently accessed data, thereby avoiding service interruption and maintaining search continuity.
Solution Approach 2:
The system maintains continuous search service operation throughout the index file transition. By generating warm-up queries dynamically during ongoing search operations and using a query replication mechanism that operates concurrently with normal search traffic, the system ensures uninterrupted search functionality while still achieving effective cache warming up.
2Speed
If query replication rate is increased to speed up warming up, then cache population speed improves, but I/O impact on disk increases
Solution Approach 1:
The system dynamically adjusts the query replication rate based on real-time conditions. The replication rate is not fixed but adapts to the current state of the system, allowing faster warming up when disk I/O capacity is available and reducing replication when I/O pressure is high, thereby optimizing both cache population speed and disk I/O impact.
Solution Approach 2:
The system changes the parameter of query replication rate to optimize performance. By adjusting this parameter, the system can control the balance between cache warming up speed and disk I/O load, enabling flexible adaptation to different operational conditions and resource availability.
Data Source
Figure 1A~1B
Figure 2
Figure 3
AI summary
The present disclosure provides a method, an apparatus and a search node for warming up index files during a search service. The search service is provided by at least one search node in a search system. A first index file is stored in a disk in the search node, and a first group of pages from the first index file is stored in a cache in the search node. A second index file is stored into the disk in the search node. A plurality of queries is received during the search service. At least one warm-up query is determined from the plurality of queries. A warm-up search is performed based on the second index file in response to the at least one warm-up query. A second group of pages in the second index file is stored into the cache based on a result of the warm-up search.