Message Broker Cluster Synchronization via Orchestration Server
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed PUB/SUB message queue systems face challenges in synchronizing message queues among message brokers, leading to inconsistencies and reduced throughput, as they often rely on fixed brokers and lack parallelism in data transmission.
Innovation Solution
A method and system that utilize a message broker cluster with buffer and data queues, where an orchestration server selects a message broker for synchronization, sets a data read-lock, and copies data contents from all brokers to ensure consistency across the cluster, allowing clients to select brokers based on network conditions for high throughput and parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data synchronization is performed among message brokers, then data consistency is improved, but transmission throughput is reduced
Solution Approach 1:
The patent divides the message queue into two separate queues: a buffer queue for receiving and temporarily storing messages, and a data queue for maintaining consistent data across brokers. This segmentation allows the buffer queue to handle high-speed message ingestion while the data queue ensures consistency through controlled synchronization, resolving the contradiction between throughput and consistency.
Solution Approach 2:
The patent implements preliminary synchronization by having the orchestration server proactively manage the data queue and coordinate synchronization operations before data inconsistency occurs. The system pre-establishes synchronization mechanisms and lock protocols to ensure consistency is maintained without blocking normal high-speed message transmission operations.
2Ease of operation
If fixed brokers are used for publication and subscription, then data transmission is simplified, but parallelism and adaptability are reduced
Solution Approach 1:
The patent introduces dynamic broker selection capabilities where publishers and subscribers can choose from multiple brokers rather than being fixed to a single broker. The system dynamically assigns and reassigns brokers based on real-time conditions, enabling parallel operations across multiple brokers while maintaining operational simplicity through automated orchestration.
Solution Approach 2:
The patent creates a universal message queue system where any broker can serve any publisher or subscriber dynamically. The buffer queue and data queue structure provides multi-functionality, allowing the same infrastructure to handle both high-throughput message routing and consistent data synchronization across all brokers, eliminating the need for fixed dedicated paths.
3Reliability
If data read-lock is set on data queue, then data consistency is ensured, but continuous publication is blocked
Solution Approach 1:
The patent separates publication operations (buffer queue) from data consistency operations (data queue). The buffer queue continues to receive and temporarily hold messages even when the data queue has a read-lock active, allowing continuous publication while ensuring consistency is maintained in the data queue without blocking incoming message flow.
Solution Approach 2:
The buffer queue acts as an intermediary between message sources and the data queue. It decouples the continuous stream of incoming messages from the consistency-critical data queue, allowing messages to be accumulated in the buffer queue while the data queue maintains locks to ensure consistency, thus preventing publication blockage.
Data Source
AI summary
A system and method for synchronizing publication and subscription of message queues are provided. The method includes: providing a message broker cluster including a plurality of message brokers each having a buffer queue and a data queue; as any one of the message brokers requests a synchronization requirement, selecting, by an orchestration server, one of the message brokers in the message broker cluster to perform data synchronization; setting a data read-lock to the data queue of the selected message broker sequentially to write data contents in the buffer queues of all of the plurality of message brokers into the data queue of the selected message broker; and copying the complete data contents in the data queue of the selected message broker to the data queues of the other message brokers, allowing the data contents in the data queues of each of the message brokers to be consistent.