Aggregate Event Records for Time-Windowed Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large networked systems with vast quantities of event-related data, storing and querying event data becomes prohibitively large, leading to inefficiencies in retrieving aggregated data over time windows, as existing methods fail to preserve historical context while updating aggregated records.
Innovation Solution
A computer-implemented method that processes streamed event data by creating and updating aggregate event records within defined time windows, preserving historical data by generating new records with updated timestamps, and providing aggregated data to client devices based on query parameters, including start and end times, and snapshot times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all event data is stored in detail, then data completeness is improved, but query efficiency deteriorates due to prohibitively large storage requirements
Solution Approach 1:
The patent segments event data by dividing it into time windows and creating separate aggregate records for each window. This segmentation allows the system to store summarized data for each time period rather than all individual events, reducing overall storage requirements while maintaining the ability to retrieve complete information for any specific time window when needed.
Solution Approach 2:
The system performs preliminary aggregation of event data into time-windowed summary records before queries are executed. By pre-computing and storing aggregate statistics (such as event counts, sums, or other aggregations) for each time window, the system prepares data in advance so that queries can be answered quickly by retrieving pre-aggregated results rather than computing them on-demand from raw events.
2Measurement precision
If aggregate event records are updated frequently, then data accuracy is improved, but historical context is lost
Solution Approach 1:
When event data arrives that belongs to a previous time window, the system creates a copy of the existing aggregate record for that window, updates the copy with the new event information, and stores it as a new version. This copying mechanism allows the system to maintain updated accurate records while preserving the original historical records unchanged, thus maintaining both data accuracy and historical context.
Solution Approach 2:
The patent implements a dynamic record system where aggregate event records can be updated and versioned over time. Each aggregate record associated with a time window can be dynamically updated when new events are received, with the system managing multiple versions of records as needed. This dynamic approach allows the data to remain current while preserving historical states.
3Measurement precision
If aggregate event records are updated retroactively, then data accuracy is improved, but system complexity increases
Solution Approach 1:
The patent extracts the update logic into a separate, dedicated component that handles retroactive updates independently from the main event processing flow. When events arrive late or need to update previous time windows, a specific update mechanism retrieves relevant aggregate records, applies updates, and manages versioning in isolation. This extraction of the update function reduces complexity by separating concerns and providing a focused approach to handling retroactive updates.
Data Source
AI summary
Methods and systems for processing real-time event data by aggregating event data over various time windows and storing aggregate event data records containing the aggregated data. The method may include receiving event data at a current time, determining that the event data relates to a time prior to the beginning of a current time window, and identifying, based on the time, one or more stored aggregate event records each associated with a respective previous time window. The method may include creating, for each of the identified one or more stored aggregate event records, a new aggregate event record containing updated aggregate event data adjusted by the event data, and storing each new aggregate event record in association with its respective previous time window and a creation timestamp reflecting the current time.