Bucket Search Metric Rebalancing Across Peer Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing bucket rebalancing mechanisms in distributed computing systems fail to account for the disparity in processing requirements due to varying search frequencies across peer nodes, leading to inadequate load balancing and increased latency.
Innovation Solution
Implementing bucket search metrics to measure the actual searches performed on individual buckets, aggregate these metrics across peer nodes, and use deviation from the average to rebalance workload, ensuring even distribution of processing loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional bucket rebalancing mechanisms are used that only consider the number of buckets or amount of data stored, then the rebalancing process is simple to implement, but the processing load distribution remains uneven leading to increased latency
Solution Approach 1:
The patent changes the rebalancing parameter from simple bucket count or data volume to composite search metrics including search frequency, data size, and processing requirements. This allows the system to account for actual processing loads rather than just storage distribution, reducing latency while maintaining manageable complexity through automated metric collection and evaluation.
2Productivity
If bucket rebalancing is performed frequently to maintain even distribution, then load balancing improves, but system overhead and complexity increase
Solution Approach 1:
The patent implements a feedback mechanism where search metrics are continuously collected from peer nodes, evaluated against target distributions, and used to trigger rebalancing operations only when necessary. This feedback loop maintains load balancing efficiency by responding to actual system conditions while avoiding unnecessary rebalancing operations that would increase complexity and overhead.
Solution Approach 2:
The rebalancing system is made dynamic by adjusting rebalancing thresholds and frequencies based on system conditions. The patent allows the rebalancing behavior to adapt to changing search patterns and system states, maintaining efficiency without requiring complex static configurations or overly frequent operations.
3Measurement precision
If search metrics are collected and processed for every bucket across all peer nodes, then accurate load balancing is achieved, but measurement and processing complexity increases
Solution Approach 1:
The patent creates a universal metric collection framework that operates across all peer nodes using standardized procedures. The same metric collection, evaluation, and rebalancing logic is applied uniformly throughout the distributed system, achieving accurate measurement without requiring node-specific complex implementations. This multi-functional approach handles metric collection, evaluation, and rebalancing decisions through a consistent system-wide process.
Data Source
AI summary
Bucket search metric-based rebalancing across peers includes obtaining bucket search metrics of searches performed on buckets located on peer nodes. The bucket search metrics are aggregated on a per peer node basis to obtain an aggregated bucket search metric for each peer node. An average aggregated bucket search metric is calculated across the peer nodes. The first subset of the peer nodes having the aggregated bucket search metric greater than the average aggregated bucket search metric is identified. The first subset includes a source peer node of the first subset having a deviation of an aggregated bucket search metric of the source peer node from the average aggregated bucket search metric. One or more buckets on the source peer node are moved from the source peer node to at least one target peer node of a second subset of the peer nodes.