Multi-Message Queue Routing for Low-Latency and High-Throughput Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current message queue systems face challenges in balancing high performance and low cost, with existing systems like Apache Kafka and Apache RocketMQ having limitations in throughput and storage costs due to their unique architecture designs, which affect processing efficiency and resource overhead as service data volumes grow.
Innovation Solution
A hybrid storage message queue system is introduced, combining a data hierarchical storage method with multiple message queues, utilizing the design advantages of different queue systems to provide a system with higher performance and lower cost by selectively using data access interfaces based on service scenarios, such as low-latency or high-throughput modes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single message queue system architecture is used, then system simplicity is maintained, but processing efficiency and adaptability to different service scenarios deteriorate
Solution Approach 1:
The system divides the message queue infrastructure into multiple independent message queue systems, each optimized for specific service scenarios. This segmentation allows different queue systems to handle different types of requests (instant vs. intensive) with specialized configurations, thereby improving overall processing efficiency while maintaining manageable complexity through clear separation of concerns
Solution Approach 2:
The message queue selection mechanism provides a universal interface that can route to multiple different message queue systems based on service scenario requirements. This multi-functionality allows the same system architecture to serve both instant service scenarios and intensive service scenarios efficiently, improving productivity without requiring scenario-specific code paths
2Speed
If high-performance message queue systems are used, then data processing speed is improved, but storage costs increase
Solution Approach 1:
The system applies local quality by matching message queue characteristics to specific service scenario requirements. High-performance, high-cost message queue systems are used locally for instant service scenarios where speed is critical, while cost-effective message queue systems are used for intensive service scenarios where throughput is prioritized over latency. This localized optimization improves data processing speed for time-sensitive operations while controlling storage costs through appropriate system selection
3Adaptability or versatility
If multiple heterogeneous message queue systems are combined, then adaptability to different service scenarios is improved, but system complexity increases
Solution Approach 1:
The system introduces a message queue selection mechanism as an intermediary layer between the application and multiple heterogeneous message queue systems. This mediator handles the complexity of routing decisions, scenario classification, and system selection, thereby improving adaptability to different service scenarios while shielding the application from the underlying architectural complexity. The intermediary absorbs the complexity burden through centralized management logic
Data Source
AI summary
Embodiments of the present disclosure provide a data processing method and a device based on a multi-message queue system, the method includes: receiving a data read request, where the data read request is used to request to read first target data; if the data read request belongs to an instant service scenario, reading the first target data from a first message queue system in a heterogeneous multi-message queue system by using a data access interface of the first message queue system to respond; and if the data read request belongs to an intensive service scenario, reading the first target data from a second message queue system in the heterogeneous multi-message queue system by using a data access interface of the second message queue system to respond.