In-Memory Buffer Service for High-Volume Event Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-volume event systems face latency issues when processing and storing large volumes of events, such as those generated by applications like Apache HBase, due to conventional storage methods that are inefficient and result in unacceptable levels of latency.
Innovation Solution
An in-memory buffer service is implemented on a multi-tenant application server, providing a low-latency API for writing event records to a data store, using a 'store and forward' mechanism with bounded buffers that temporarily store events in memory until they can be efficiently flushed to a data store, supporting concurrent writing and dynamic thread pooling for high throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional storage methods are used for high-volume event systems, then data can be stored permanently, but latency increases to unacceptable levels
Solution Approach 1:
The system segments the storage process into two distinct phases: an in-memory buffering phase for rapid data intake and a subsequent persistence phase for durable storage. This segmentation allows the system to achieve low latency during event ingestion while maintaining data reliability through eventual persistence to disk-based storage.
Solution Approach 2:
An in-memory buffer acts as an intermediary component between the event source and the persistent storage system. This intermediary absorbs the latency of disk I/O operations, allowing events to be accepted rapidly into memory while being flushed to persistent storage asynchronously, thereby decoupling the speed of ingestion from the speed of permanent storage.
2Reliability
If data is stored immediately in a data store, then permanent storage is achieved, but processing speed decreases due to I/O operations
Solution Approach 1:
The system performs preliminary storage action by writing data to in-memory buffers before committing to persistent storage. This preliminary action in volatile memory allows rapid data capture without the overhead of immediate disk I/O, and the data is subsequently persisted in batches or asynchronously, maintaining both speed and reliability.
Solution Approach 2:
The in-memory buffer enables continuous data acceptance and processing without interruption from I/O operations. The buffer continuously accepts events at high speed while separate persistence operations occur in the background, ensuring that the useful action of data ingestion continues uninterrupted while data is gradually persisted to storage.
3Productivity
If buffer size is increased to handle high-volume events, then throughput improves, but memory consumption increases
Solution Approach 1:
The buffer implementation uses dynamic data structures that can adapt their memory allocation based on the current event volume and system conditions. The buffer can grow and shrink as needed, allowing the system to handle variable throughput requirements while optimizing memory consumption rather than allocating fixed large memory spaces.
Solution Approach 2:
The system can dynamically adjust buffer parameters such as capacity limits, flush thresholds, and retention policies based on workload conditions. When memory pressure is detected, parameters are changed to reduce buffer size or accelerate flushing to persistent storage, thereby maintaining throughput while controlling memory usage.
Data Source
AI summary
A capture service running on an application server receives events from a client application running on an application server to be stored in a data store and stores the events in an in-memory bounded buffer on the application server, the in-memory bounded buffer comprising a plurality of single-threaded segments, the capture service to write events to each segment in parallel. The in-memory bounded buffer provides a notification to a buffer flush regulator when a number of events stored in the in-memory bounded buffer reaches a predefined limit. The in-memory bounded buffer receive a request to flush the events in the in-memory bounded buffer from a consumer executor service. The consumer executor service consumes the events in the in-memory bounded buffer using a dynamically sized thread pool of consumer threads to read the segments of the bounded buffer in parallel, wherein consuming the events comprises writing the events directly to the data store.