In-Memory Buffer Service with Temporary Events File Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current in-memory buffer services face challenges in handling high-volume event data with high throughput, leading to latency and reliability issues, particularly when the data store is unavailable or experiences high loads, resulting in potential data loss and unacceptable levels of latency.
Innovation Solution
The implementation of a temporary events file storage system (TEFSS) that temporarily stores events as files, allowing for asynchronous writing to a data store, which enhances reliability and reduces latency by decoupling event capture from data store availability and providing a scalable solution for high-volume event handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If events are stored directly in an in-memory buffer service, then write latency is reduced, but reliability deteriorates when the data store is unavailable
Solution Approach 1:
The system performs preliminary action by writing events to the data store asynchronously in the background, allowing the capture service to immediately acknowledge event receipt. This decoupling ensures low write latency while maintaining reliability through persistent storage, as events are safely stored even when the data store experiences high load or temporary unavailability.
2Productivity
If high-volume events are processed through a traditional buffer service, then throughput is limited, but latency increases
Solution Approach 1:
The system implements continuity of useful action by maintaining an unbroken flow of event processing through asynchronous operations. The capture service continuously receives and acknowledges events immediately, while a separate consumer service continuously processes and stores events in the data store, ensuring high throughput without blocking latency-critical operations.
3Stability of the object's composition
If the data store is tightly coupled with the buffer service, then data consistency is maintained, but system scalability deteriorates
Solution Approach 1:
The system introduces an intermediary asynchronous processing layer between the capture service and the data store. This mediator pattern allows the buffer service to scale independently from the data store while maintaining data consistency through reliable asynchronous processing, enabling the system to handle varying loads and expand capacity flexibly.
Data Source
AI summary
Computer-implemented methods and systems are provided for writing events to a data store. An application server generates events, the data store that stores the events, and a temporary events file storage system (TEFSS) temporarily stores groups of events as events files. When events are unable to be written directly to the data store, an indirect events writer is invoked that includes event capture threads each being configured to generate a particular events file, and write it to the TEFSS. Each events file includes a plurality of events flushed from an in-memory buffer service. An events file uploader service reads the events file(s) from the TEFSS, and then writes the events from each of the events files to the data store.