Event Pipeline Controller for Systems of Record
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems experience high latency and inefficiencies in processing events, particularly in systems of record that handle a high volume of unique events related to product development and services.
Innovation Solution
The implementation of an event pipeline with a controller, service, and publisher-subscriber unit that allows for direct and indirect transmission of events, ensuring reliable and real-time delivery to interested receivers, while also employing deduplication to reduce redundancy and unnecessary traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional event processing systems are used, then system simplicity is maintained, but latency is high and processing efficiency is poor
Solution Approach 1:
The system is divided into distinct functional components: event sources that generate events, an event pipeline that transports events, and event consumers that process events. This segmentation allows each component to be optimized independently, improving overall event processing efficiency while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
An event pipeline acts as an intermediary between event sources and event consumers. This mediator component enables direct event transmission from sources to consumers without requiring complex point-to-point connections, thereby improving processing efficiency while keeping the system architecture simple and standardized.
2Reliability
If redundant event transmission is allowed, then reliability is improved, but network traffic increases and efficiency decreases
Solution Approach 1:
The system implements feedback mechanisms where event consumers can acknowledge receipt of events, and the event pipeline can track event delivery status. This feedback loop ensures reliable event delivery by confirming successful transmission while avoiding redundant retransmission of already-delivered events, thus maintaining both reliability and processing efficiency.
3Speed
If real-time event delivery is implemented, then responsiveness is improved, but latency increases due to processing overhead
Solution Approach 1:
Event consumers establish subscriptions and registration in advance with the event pipeline before events are generated. This preliminary action allows the system to prepare event routing paths and consumer readiness states beforehand, enabling immediate event delivery when events occur without the overhead of dynamic connection establishment, thus achieving real-time delivery with minimal latency.
Data Source
AI summary
Techniques for propagation of events are described. A system for propagation of events includes a controller, a service, and a publisher-subscriber (pub-sub) unit. The controller may route a request for modification of an object to a service. The modification of the object generates an event and the object corresponds to data in a system of record. The service corresponds to the event generated. The service processes the request for the modification of the object and transmit the event at least one of: directly or indirectly to a publisher-subscriber (pub-sub) unit. The pub-sub unit publishes the received event through the controller to a plurality of receivers. The plurality of receivers corresponds to receivers that are interested about the event.