In-Memory Aggregated Data for Real-Time Event Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Highly distributed computing environments face delays in tracking and monitoring automated and user-initiated events due to the complexity and vast amounts of data, leading to impractical delays in system monitoring, which is costly in terms of time and money, and hinders timely system changes.
Innovation Solution
A system that captures, aggregates, and stores data in real-time or near-real time, eliminating the need for inefficient polling and long-running queries, and generates dashboards for real-time display of product utilization, KPIs, and system events, enabling faster troubleshooting and issue resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional polling and long-running queries are used to track system events, then measurement accuracy is improved, but response time deteriorates significantly
Solution Approach 1:
The system pre-processes and aggregates event data as it arrives, maintaining ready-to-query summaries in memory. When a monitoring request occurs, the pre-computed aggregated data is immediately available, eliminating the need for time-consuming queries over raw event streams while maintaining accurate event tracking.
Solution Approach 2:
An intermediary aggregation layer is introduced between raw event sources and monitoring queries. This layer continuously processes incoming events, computes aggregated metrics, and stores them in an optimized format, serving as a buffer that enables fast querying without sacrificing measurement precision.
2Measurement precision
If comprehensive event data is collected and stored for analysis, then measurement precision is improved, but system complexity increases
Solution Approach 1:
The system segments event data processing into distinct stages: raw event collection, aggregation by predefined dimensions (time, source, type), and selective storage. This segmentation allows the system to maintain comprehensive event information while organizing it in a structured manner that reduces processing complexity and enables efficient querying.
Solution Approach 2:
The system transforms raw event data into aggregated parameters (counts, averages, time-series metrics) that capture essential event characteristics in a condensed form. This parameter transformation maintains analytical precision while significantly reducing data volume and processing complexity.
3Productivity
If real-time data processing is implemented, then productivity is improved, but energy consumption increases
Solution Approach 1:
The system implements partial real-time processing by continuously aggregating events in the background but only performing full analysis and storage when triggered by monitoring queries or threshold conditions. This approach maintains productivity benefits of real-time processing while reducing energy consumption by avoiding continuous full-scale analysis.
Solution Approach 2:
The system changes the state of data from raw individual events to aggregated summaries, reducing the computational energy required for processing. By maintaining aggregated parameters in memory with minimal processing overhead, the system achieves real-time monitoring capability with substantially lower energy consumption compared to processing complete event streams.
Data Source
AI summary
Aspects of the disclosure track and monitor automated system events, user-initiated events, and outcomes associated with system events and user-initiated events for providing real-time or near real-time measurements and insights. According to examples, a system is provided that is operative or configured to collect specific events and associated data associated with events that occur within various systems and products. The data is captured, aggregated, and stored in in-memory data storage. Accordingly, the data can be represented and displayed in a dashboard interface in real-time or near real-time. Accordingly, inefficient polling, long-running queries, and scheduled tasks can be eliminated.