Event-Driven Rolling Aggregates for Report Generation Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the volume of event data grows, generating useful reports from stored event information becomes increasingly resource-intensive and time-consuming, particularly when users need to aggregate data from various events, requiring multiple database queries that consume significant processing resources.
Innovation Solution
A system that aggregates key values associated with events as they are processed, maintaining key-value mappings and key logs to quickly generate information by updating and recording aggregate values, allowing for efficient reporting through reduced operational needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If event information is stored for later parsing and report generation, then comprehensive data availability is improved, but computing power and processing time requirements increase significantly
Solution Approach 1:
The system performs preliminary aggregation of event data into key-value mappings during event processing, so that when reports are needed, the data is already pre-computed and ready for quick retrieval. This eliminates the need to process raw event data from scratch during report generation.
Solution Approach 2:
The system creates aggregated copies of event data in the form of key-value mappings that summarize event characteristics. These aggregated copies are stored and can be quickly retrieved for reporting, avoiding the need to re-process the original raw event data.
2Adaptability or versatility
If multiple database queries are performed to aggregate data from various events, then comprehensive reporting capability is improved, but processing resources and time consumption increase
Solution Approach 1:
The system pre-aggregates event data into key-value mappings during event processing, so that reporting operations can directly query these pre-computed aggregates rather than performing multiple database queries to aggregate raw data. This significantly reduces processing resources during reporting.
Solution Approach 2:
The system extracts and stores aggregated data in a separate key-value mapping structure that is optimized for reporting operations. This allows reporting to access pre-extracted information directly, avoiding the need to query and aggregate from the raw event database.
3Loss of information
If raw event data is stored and queried for aggregation, then data completeness is improved, but the number of operations and processing time increase
Solution Approach 1:
The system performs preliminary aggregation of event data into key-value mappings during event processing. When reports are needed, the system queries these pre-aggregated values directly, significantly reducing processing time while maintaining data completeness through the key-value mapping structure.
Solution Approach 2:
The system creates aggregated copies of event data in key-value mappings that preserve the necessary information for reporting. These copies are stored and can be quickly retrieved, avoiding the time-consuming process of querying and aggregating from raw event data while maintaining data completeness.
Data Source
AI summary
Described herein are systems and methods in which key values associated with events are aggregated as those events are processed by an event processing engine. In some embodiments, the system maintains a number of key-value mappings associated with a plurality of key values to be updated. Each key value may be associated with a key log that can be accessed to quickly generate useful information while minimizing the number of operations needed to generate that information. The key log may be updated each time that the associated key value is updated in order to maintain a running history of key-value updates.