Log Server Cluster for Reliable Asynchronous Message Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for delivering high volumes of asynchronous messages over unreliable networks, such as those used in airline billing transactions, face challenges in maintaining reliability and throughput due to message loss and lack of fault tolerance in log servers.
Innovation Solution
A method and system that utilize a cluster of log servers to receive and process application messages and control messages, where control messages are used to synchronize file splitting and select the most reliable application data file for post-processing by comparing control files across log servers, ensuring high reliability and performance even over unreliable networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous message transmission is used over unreliable networks to increase throughput, then message delivery speed improves, but message loss increases
Solution Approach 1:
The system segments message delivery into two independent phases: asynchronous transmission phase (for high throughput) and synchronous verification phase (for reliability). Control messages are sent asynchronously to multiple log servers to maximize throughput, then a coordination mechanism synchronizes the verification phase where the sending application waits for acknowledgments to ensure reliable delivery.
Solution Approach 2:
Control messages act as intermediaries between the application and log servers. These control messages carry transaction identifiers and are used to track message delivery status, enable verification of successful delivery, and coordinate the transition between asynchronous and synchronous phases without directly blocking the high-volume message stream.
2Reliability
If multiple log servers are used to improve reliability through redundancy, then fault tolerance improves, but system complexity increases
Solution Approach 1:
Multiple log servers are merged into a coordinated redundant system where each server independently stores message copies. The system combines the asynchronous message handling capability of individual servers with a centralized coordination mechanism that manages verification and failure recovery, achieving fault tolerance without requiring complex inter-server communication protocols.
Solution Approach 2:
Each log server independently processes incoming messages and maintains its own state, providing self-service capability. When redundancy is needed, multiple self-service servers are deployed, and the coordination mechanism simply orchestrates verification without requiring the servers to complexly interact with each other, thus limiting the increase in system complexity.
3Reliability
If control messages are sent at predetermined intervals to synchronize file splitting, then message delivery reliability improves, but network overhead increases
Solution Approach 1:
Control messages are sent at predetermined intervals rather than continuously, creating periodic synchronization points. This periodic action is sufficient to maintain reliability by ensuring file splitting is coordinated at regular checkpoints, while significantly reducing network overhead compared to continuous control message transmission. The intervals are tuned to balance reliability requirements with network efficiency.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The invention provides a computer-implemented method and a system of providing high reliability and high performance application message delivery, characterized in that it comprises the following steps performed with at least one data processor: at a plurality of log servers (203 a - b) coupled to at least an application server (201 a - c): receiving asynchronously, from the at least one application server (201 a - c), application messages containing application information; receiving asynchronously, from the at least one application server (201 a - c), control messages at a predetermined interval; at each of the plurality of log servers (203 a - b): storing the received application messages in a current application data file (209) and the received control messages in a control file (211); and upon receiving an open-close control message, closing the current application data file, storing said closed application data file and creating a new application data file as the current application data file; comparing the control files of the plurality of log servers (203 a - b) for a given interval; and based on this comparison, determining from among a plurality of application data files (209) from each of the log servers (203 a - b), an application data file as a best-candidate for a given interval and forwarding the best-candidate file for post-processing.