Watermark-Based Message Queue for Efficient Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing message queue systems face challenges in efficiently identifying and delivering messages to user devices, leading to increased memory seeking times, hard drive fatigue, and power consumption, while also risking undelivered messages due to unstable network connections.
Innovation Solution
The system employs a watermark-based message queue mechanism, where a watermark identifies the oldest message in the queue at the start of each connection session, allowing for efficient scanning and delivery of messages with timestamps newer than or equal to the watermark, while also managing acknowledged and unacknowledged messages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system scans through the entire message queue to identify messages for delivery, then all messages can be found, but memory seeking time increases and hard drive fatigue occurs
Solution Approach 1:
The system performs preliminary action by maintaining a watermark that records the position of the oldest unacknowledged message from previous connection sessions. This pre-established marker eliminates the need to scan from the beginning of the queue each time, allowing the system to start scanning from the watermark position and thus reducing memory seeking time while ensuring complete message delivery.
Solution Approach 2:
The system extracts the essential information (oldest message position) from the entire message queue history and represents it through a compact watermark. This extracted marker allows the system to ignore the vast portion of already-delivered messages and focus only on the relevant unacknowledged messages, significantly reducing the scanning range and associated time costs.
2Device complexity
If the system uses traditional message queue scanning methods, then message identification is simple, but hard drive fatigue and power consumption increase
Solution Approach 1:
The watermark is updated in advance during each connection session to record the position of the oldest unacknowledged message. This preliminary update creates a persistent marker that guides future scanning operations, reducing the range of data that needs to be read from the hard drive and thereby lowering power consumption while maintaining simple message identification logic.
3Reliability
If the system scans from the beginning of the message queue for each connection session, then all messages are identified, but the system can service fewer user devices
Solution Approach 1:
The system performs the preliminary action of tracking and storing the watermark position from previous sessions. This allows each connection session to independently and efficiently scan only from the appropriate watermark position, ensuring reliable message delivery to all users without the scanning time penalty that would limit the number of devices that can be serviced simultaneously.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a watermark-based message queue. One of the methods includes receiving a first connection request for messages associated with a user device. A first connection session is established with the user device. A message queue of messages associated with the user device is identified, each message in the message queue is associated with a respective timestamp, and the message queue is associated with a current watermark that identifies a first timestamp. An oldest message in the message queue at the time the first connection session was established is identified. An updated watermark that identifies a second timestamp associated with the oldest message is associated with the message queue. One or more messages that have a timestamp newer than or equal to the first timestamp identified by the current watermark is provided to the user device.