Distributed Server Message Durability via Transaction Log Retransmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed server systems, achieving durability while maintaining availability is challenging due to the tension between consistency and availability, often resulting in message loss or dropout before being written to storage, especially in high-volume, latency-sensitive systems.
Innovation Solution
Implementing an availability-oriented durability technique by generating a unique identifier for messages at the entry point of a message processing stream, storing them in a transaction log, and retransmitting if not successfully processed and persisted, using a memory-mapped approach to enhance processing speed and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If messages are written to storage at the end of a message processing stream to reduce latency, then availability is improved, but durability deteriorates due to potential message loss or dropout
Solution Approach 1:
The patent assigns unique identifiers to messages at the entry point of the message processing stream before they are processed through multiple application layers. This preliminary action enables the system to track and retransmit messages that may be lost during processing, resolving the contradiction by maintaining durability guarantees while allowing asynchronous processing that improves latency.
Solution Approach 2:
The patent implements a feedback mechanism where the system monitors whether messages have been successfully persisted to storage and triggers retransmission if not. This feedback loop ensures durability is maintained even when messages are processed asynchronously at the end of the stream, allowing the system to prioritize latency while recovering from any message loss.
2Productivity
If messages are processed through multiple application layers before being written to storage, then availability is improved, but the potential for message loss increases
Solution Approach 1:
The patent creates a copy of the message payload stored in the transaction log using the assigned unique identifier. This copying mechanism allows the original message to be processed through multiple application layers for high throughput while maintaining a recoverable copy that ensures message delivery guarantees, thus resolving the contradiction between productivity and reliability.
3Reliability
If durability is prioritized by writing messages to storage immediately, then message loss is prevented, but availability and processing speed decrease
Solution Approach 1:
The patent performs preliminary actions of assigning unique identifiers and storing message payloads in transaction logs without immediately persisting them to durable storage. This allows messages to be processed quickly through the system while maintaining the capability to recover and persist them later, thus achieving high processing speed while preserving durability through on-demand recovery.
Data Source
AI summary
Disclosed are systems, methods, and non-transitory computer-readable media for an availability oriented durability technique for distributed server systems. A server that is an entry point for a message processing stream receives a message from a client device, and in response, generating a unique identifier for the message, and adds an entry in a transaction log that including the message and the unique identifier for the message. The server appends the unique identifier to the message, and transmits the message to a second server positioned downstream from the server in the message processing stream. If the server determines that the message has not been processed through the message processing stream, the server accesses the message from the transaction log, appends the unique identifier to the first message, and re-transmitting the message to the second server positioned downstream from the server in the message processing stream.