In-Memory Buffer Framework for Real-Time Analytics Metric Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-tenant database systems face challenges in efficiently generating and storing real-time analytics metric data, as manual querying of raw event data can overwhelm computing resources and delay processing, while existing solutions often generate aggregate metrics after event capture, rather than in real-time.
Innovation Solution
An in-memory buffer service consumer framework is employed to generate and store real-time analytics metric data by using a system with concurrent bounded buffers and buffer flush regulators, allowing for parallel processing and configurable workflow steps, including synchronous, asynchronous, or time-bounded executors, to derive metrics concurrently with event capture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual querying of raw event data is used to generate analytics metrics, then metric data can be obtained, but computing resources are overwhelmed and processing is delayed
Solution Approach 1:
The system performs preliminary actions by capturing and buffering raw event data in real-time as it occurs, rather than querying later. The in-memory buffer service stores events concurrently with their generation, preparing the data structure in advance so that metric generation can occur immediately without resource-intensive retroactive querying.
Solution Approach 2:
The in-memory buffer service acts as an intermediary between event capture and metric generation. It receives raw events from multiple sources, maintains them in bounded buffers, and provides them to metric generation consumers, thereby decoupling the resource-intensive querying operation from the event capture process and enabling parallel execution.
2Reliability
If aggregate metrics are generated after event capture, then resource overload is avoided, but real-time analytics capability is lost
Solution Approach 1:
The system maintains continuous useful action by generating metrics concurrently with event capture through parallel consumer threads. Multiple consumers process different aspects of event data simultaneously - some generating aggregate metrics, others deriving real-time analytics - ensuring that metric generation is an ongoing continuous process rather than a delayed batch operation.
Solution Approach 2:
The system employs dynamic parallel processing where multiple consumer threads adaptively process events in real-time. The bounded buffers dynamically manage event queues, and consumers can be configured with different processing strategies (synchronous, asynchronous, time-bounded) to balance system stability with real-time responsiveness based on current load conditions.
3Productivity
If multiple consumers process events in parallel, then processing efficiency increases, but system complexity increases
Solution Approach 1:
The system segments the metric generation process into multiple independent consumer threads, each handling specific processing tasks. The in-memory buffer service divides event streams into bounded buffers that can be consumed in parallel, allowing different consumers to process different portions of the event data simultaneously without interfering with each other, thereby increasing throughput while maintaining manageable complexity through clear separation of concerns.
Data Source
AI summary
In an example, a processing system of a database system may categorize event data taken from logged interactions of users with a multi-tenant information system to provide a metric. Event roll-up aggregate metrics used to provide the metric may be generated in connection with event capture. The processing system of the database system may periodically calculate the metric for a particular one of the tenants, and electronically store the periodically calculated metrics for accessing responsive to a query of the particular tenant.