Transparent Sharding Across Messaging Brokers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional message broker configurations in database systems, where brokers are set up as active/passive pairs, lead to underutilization of resources during peak loads, as the passive broker remains idle most of the time, resulting in performance compromises and inefficient equipment usage.
Innovation Solution
Implementing a broker sharding service that dynamically distributes message queues across multiple brokers, allowing for seamless switching between sharding strategies such as RAC node-based and active/passive sharding, to ensure scalability and high availability, and abstracting implementation details via APIs for flexible resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If brokers are configured as active/passive pairs with queues assigned to a single active broker, then system reliability is improved through failover capability, but resource utilization deteriorates as the passive broker remains idle most of the time
Solution Approach 1:
The patent segments the queue assignments across multiple brokers using sharding strategies. Instead of assigning all queues to a single active broker, the system divides queues into shards and distributes them across multiple brokers, allowing multiple brokers to simultaneously handle traffic and reducing idle time while maintaining reliability through distributed architecture
Solution Approach 2:
The patent implements dynamic sharding strategies that can switch between different modes (e.g., RAC node-based sharding, active/passive sharding, round-robin sharding) based on system conditions and load requirements. This dynamic adaptation allows the system to optimize resource utilization while maintaining reliability under varying operational scenarios
2Device complexity
If all message queues are assigned to a single active broker, then configuration simplicity is improved, but system performance deteriorates during peak loads due to the broker becoming a bottleneck
Solution Approach 1:
The patent segments the message broker system into multiple sharded brokers, each handling a subset of queues. This segmentation distributes the processing load during peak periods, preventing any single broker from becoming a performance bottleneck while maintaining manageable configuration through automated sharding management
Solution Approach 2:
The patent creates a universal sharding service that can operate with multiple sharding strategies and adapt to different load conditions. This multi-functional approach allows the system to automatically scale and distribute traffic across brokers, improving performance during peak loads while maintaining simplicity through centralized management
3Reliability
If a passive broker is maintained for failover purposes, then system availability is improved, but equipment usage efficiency deteriorates due to the broker remaining unused most of the time
Solution Approach 1:
The patent transforms the static active/passive broker model into a dynamic sharded architecture where brokers can transition between roles based on load conditions. Multiple brokers remain actively utilized through sharding, and failover capability is maintained through the distributed nature of the sharded queues, thereby improving equipment utilization while preserving availability
Solution Approach 2:
The sharding service automatically manages the distribution and reassignment of queues across brokers without requiring manual intervention. When broker failures occur, the system self-adjusts by redistributing shards to remaining brokers, maintaining availability while ensuring all brokers contribute to system capacity during normal operation
Data Source
AI summary
Embodiments regard transparent sharding of traffic across messaging brokers. An embodiment of a method of sharding message traffic across messaging brokers includes providing a message sharding service for message sharding among a plurality of registered message brokers, wherein providing the message sharding service includes applying a current message sharding strategy; and sharding messages among the registered message brokers according to the current message sharding strategy, including registering the message brokers and assigning message queues to registered message brokers according to the current message sharding strategy; wherein the current message sharding strategy is one of a plurality of message sharding strategies.