Event Stream Consumer Acknowledgment Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage system technologies face challenges in efficiently determining and transmitting system updates based on object mutation events, leading to potential data inconsistencies and inaccuracies in system information management, particularly in cloud environments where multiple tenants require isolated data management.
Innovation Solution
A system information management architecture that utilizes object mutation events published in an event stream to determine and transmit updates, involving a publisher, stage one and stage two consumers, and a container service to ensure accurate and reliable system updates by re-processing and re-transmitting updates when acknowledgments are missed, ensuring an 'exactly-once' roll-up mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If system updates are transmitted based on object mutation events without acknowledgment verification, then transmission speed is improved, but data consistency and reliability deteriorate
Solution Approach 1:
The system implements an acknowledgment mechanism where the container service sends acknowledgments back to consumers when updates are successfully applied. This feedback loop enables the consumer to verify whether updates were received and processed correctly, resolving the contradiction by allowing fast transmission while maintaining reliability through verification.
Solution Approach 2:
The consumer stores update information in memory before transmission confirmation, preparing for potential retransmission. This preliminary action ensures that if an acknowledgment is not received, the system can quickly retransmit the same updates without loss, maintaining both speed and reliability.
2Reliability
If the system re-transmits updates upon missed acknowledgments, then data reliability is improved, but transmission efficiency deteriorates
Solution Approach 1:
The system implements selective retransmission rather than continuous retransmission. Updates are re-transmitted only when acknowledgments are not received within a timeout period, avoiding unnecessary retransmissions that would reduce efficiency. This partial action approach maintains reliability by ensuring critical updates are delivered while preserving overall transmission efficiency.
Solution Approach 2:
The consumer maintains copies of update information in memory that can be quickly retransmitted. Instead of re-generating updates, the system uses stored copies, which speeds up the retransmission process and minimizes the impact on overall transmission efficiency while ensuring reliable delivery.
3Loss of information
If the system processes all events from the event stream, then completeness of system information is improved, but processing time and resource consumption deteriorate
Solution Approach 1:
The system extracts and processes only the specific subset of events relevant to a particular container from the overall event stream. By filtering events based on container identifiers and update types, the system ensures completeness of relevant information while reducing processing time by excluding unrelated events.
Solution Approach 2:
The event processing is segmented by container, with each consumer instance handling specific containers. This segmentation allows parallel processing of different container events, improving overall throughput while ensuring each container receives complete and accurate update information without interfering with other containers.
Data Source
AI summary
A consumer transmits a set of system updates corresponding to an initial set of events retrieved from an event stream. The consumer determines that acknowledgement of the transmission of the initial set of events has not been received. The consumer executes another retrieval operation to retrieve events from the event stream which results in retrieving an updated set of events. The updated set of events includes the initial set of events and one or more additional events. The consumer trims the updated set of events to remove the one or more additional events and obtains the initial set of events. The consumer re-determines and re-transmits the system updates corresponding to the initial set of events that were obtained by trimming the updated set of events.