Shard Distribution and Query Routing for Distributed Search Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing shard-based search systems face inefficiencies and resource wastage due to incomplete query results when a search node fails, despite using replication for redundancy, as they can only maintain full query capacity at the cost of reduced concurrent queries or accepting partial results.
Innovation Solution
Distributing multiple shards across nodes, ensuring each node has at least two distinct shards, and implementing a query routing policy that optimizes load balancing and redundancy, allowing for high shard availability even with node failures, thus maintaining optimal query processing capacity and minimizing resource waste.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If shards are distributed across nodes with one shard per node, then query routing is simple, but system reliability deteriorates when nodes fail
Solution Approach 1:
The patent divides the index into multiple shards (m shards where m >= n*2) and distributes them across n nodes, with each node holding at least two distinct shards. This segmentation allows the system to maintain reliability even when nodes fail, as multiple shards are replicated across different nodes.
Solution Approach 2:
Each node is assigned a specific quality characteristic - holding at least two distinct shards - which differentiates it from simple one-shard-per-node distributions. This local quality enhancement at each node contributes to overall system reliability while maintaining manageable routing complexity through the query routing policy.
2Reliability
If replication is used to maintain query capacity during node failures, then system availability improves, but resource wastage increases
Solution Approach 1:
The patent implements partial replication by requiring m >= n*2 shards across n nodes, meaning each node has at least two shards but not all possible shards are replicated at every node. This partial action provides sufficient redundancy for reliability while avoiding the excessive resource consumption of full replication at all nodes.
Solution Approach 2:
The system dynamically adjusts the number of shards per node and the total number of shards based on the parameter relationship m >= n*2. This parameter change allows the system to optimize between reliability and resource efficiency by tuning the shard distribution according to specific deployment requirements.
3Productivity
If each node handles maximum requests, then query processing capacity is optimized, but system resilience to node failures deteriorates
Solution Approach 1:
The patent establishes a query routing policy in advance that accounts for node failures before they occur. The policy is designed to route queries to available nodes based on current shard locations, ensuring that query processing capacity is maintained even when nodes fail, without requiring real-time capacity optimization that would compromise resilience.
4Reliability
If multiple shards are distributed across nodes, then redundancy improves, but query routing complexity increases
Solution Approach 1:
The query routing policy incorporates feedback about current node availability and shard locations to dynamically determine query routing decisions. This feedback mechanism allows the system to maintain high redundancy through multiple shards while managing routing complexity by making routing decisions based on current system state rather than pre-configured complex rules.
Data Source
AI summary
Methods and systems are described which use load balancing to efficiently perform queries. Embodiments identify available nodes and divide an index into shards. Embodiments distribute the index across the nodes where each node has at least two distinct shards. Based on a target upper bound number of requests per node, and based on which shards exist at each node, embodiments identify a query routing policy which also includes accounting for node failure. Embodiments route a query based on the identified query routing policy.