Brokerless Inter-Process Messaging via Shared Memory and Multicast
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing intra-process messaging mechanisms face challenges when migrating to inter-process communication, particularly in ensuring reliable and totally ordered many-to-many multicast on a single node without a centralized broker, as existing solutions like UDP multicast lack guarantees and centralized brokers can create bottlenecks and failures.
Innovation Solution
A brokerless messaging system using a combination of shared memory and UDP multicast, where one plane handles data transfer and the other handles control messages, ensuring total ordering and reliability through sequence identifiers and robust futexes, allowing readers to receive messages in order even if processes fail.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If UDP multicast is used for inter-process communication, then messaging can be implemented without a centralized broker, but message delivery reliability and ordering cannot be guaranteed
Solution Approach 1:
The messaging system is segmented into two independent planes: a control plane using shared memory for coordination and ordering, and a data plane using UDP multicast for actual message transmission. This segmentation allows each plane to specialize in one function, achieving reliability through the control plane while maintaining the simplicity and brokerless architecture of the data plane.
Solution Approach 2:
Shared memory acts as an intermediary mechanism between processes, providing a common coordination space where sequence identifiers and message positions are stored. This intermediary enables total ordering and reliability guarantees without requiring a centralized broker, as processes independently access the shared memory to maintain order.
2Reliability
If a centralized broker is used to manage messaging between processes, then message delivery reliability can be ensured, but system stability deteriorates due to single point of failure
Solution Approach 1:
The centralized broker is completely removed from the system. Instead, each process independently manages its own messaging using shared memory for coordination and UDP multicast for transmission. This extraction of the broker function eliminates the single point of failure while maintaining reliability through distributed peer-to-peer communication.
Solution Approach 2:
Each process is self-sufficient and manages its own messaging without requiring a centralized broker. Processes independently handle message transmission, ordering, and delivery using the shared memory and UDP multicast mechanism, making the system resilient to individual process failures.
3Adaptability or versatility
If existing intra-process messaging mechanism is migrated to inter-process, then functionality can be extended, but code complexity increases due to migration challenges
Solution Approach 1:
The messaging system is designed to be universal, supporting both intra-process and inter-process communication through the same shared memory and UDP multicast mechanism. Existing intra-process messaging code can be reused with minimal modifications, as the same API and data structures work for both scenarios, reducing migration complexity.
Solution Approach 2:
The system uses parameter changes to adapt the messaging mechanism for different scenarios. By changing the communication mode parameter (intra-process vs. inter-process), the same underlying infrastructure serves different purposes. Sequence identifiers and message positioning parameters are adjusted to maintain compatibility across different communication types.
4Ease of operation
If readers do not read messages in the same order, then processing flexibility is improved, but message ordering guarantees are violated
Solution Approach 1:
Sequence identifiers are assigned to messages in advance before transmission. These identifiers are stored in shared memory and used by readers to determine the correct reading order. This preliminary action ensures that even though readers can process messages in any physical order, they can reconstruct the correct logical sequence using the pre-assigned identifiers.
Solution Approach 2:
The system uses feedback mechanisms where readers report their read position and sequence identifiers back to the system. This feedback allows the system to track which messages have been read and maintain ordering guarantees. Writers can use this feedback to ensure messages are transmitted in the correct sequence and to handle any ordering violations.
Data Source
AI summary
Examples described herein include systems and methods for brokerless reliable totally ordered many-to-many inter-process communication on a single node. A messaging protocol is provided that utilizes shared memory for one of the control plane and data plane, and multicast for the other plane. Readers and writers can store either control messages or message data in the shared memory, including in a ring buffer. Write access to portions of the shared memory can be controlled by a robust futex, which includes a locking mechanism that is crash recoverable. In general, the writers and readers can control the pace of communications and the crash of any process does not crash the overall messaging on the node.