Rollup Table Aggregation Preserving Temporal Data Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in reducing the size of information stored in aggregation tables while minimizing the loss of useful information, particularly in representing time/date information in rolled-up tables.
Innovation Solution
A method for aggregating data entries into roll-up entries that represent sets of values, such as time or location values, using techniques like comma list sets or bit set data items, which allow for efficient storage and retrieval of temporal distribution data, enabling the preservation of important information patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional rollup tables aggregate all detailed information into summary statistics, then storage space is reduced, but temporal distribution information is lost
Solution Approach 1:
The patent extracts only the essential temporal information (sets of time values representing when events occurred) from the detailed event data, rather than storing all raw event records. This extraction approach preserves the temporal distribution patterns needed for relationship detection while significantly reducing storage requirements compared to keeping complete event logs.
Solution Approach 2:
The patent transforms temporal distribution data from detailed event timestamps into aggregated sets of time values that capture the essential temporal patterns. This parameter transformation maintains the information needed to detect relationships between events (such as whether events occurred in sequence or within specific time windows) while reducing the data volume suitable for rollup table storage.
2Loss of information
If detailed event data is stored in full, then temporal distribution information is preserved, but storage requirements increase significantly
Solution Approach 1:
The patent extracts only the essential temporal information (sets of time values representing when events occurred) from the detailed event data, rather than storing all raw event records. This extraction approach preserves the temporal distribution patterns needed for relationship detection while significantly reducing storage requirements compared to keeping complete event logs.
3Productivity
If rollup tables store aggregated data without temporal information, then query speed improves for simple metrics, but relationship detection between events becomes impossible
Solution Approach 1:
The patent transforms temporal distribution data from detailed event timestamps into aggregated sets of time values that capture the essential temporal patterns. This parameter transformation maintains the information needed to detect relationships between events (such as whether events occurred in sequence or within specific time windows) while reducing the data volume suitable for rollup table storage.
Solution Approach 2:
The patent applies partial aggregation by retaining specific temporal information (sets of time values) within the rollup tables rather than completely aggregating all data into single summary statistics. This partial retention of detailed temporal data enables relationship detection while still providing the performance benefits of aggregation for other query types.
4Loss of information
If sets of time values are stored in rollup tables, then temporal patterns are preserved, but storage efficiency decreases compared to simple counts
Solution Approach 1:
The patent transforms temporal distribution data from detailed event timestamps into aggregated sets of time values that capture the essential temporal patterns. This parameter transformation maintains the information needed to detect relationships between events (such as whether events occurred in sequence or within specific time windows) while reducing the data volume suitable for rollup table storage.
Data Source
AI summary
Techniques for making aggregated entries in a database table which aggregate information from other entries in tables in the database system. The techniques permit the aggregated entries to contain not only metric values aggregated from the other entries by techniques such as averaging in which the individual values are lost, but also sets of individual values from the other entries. One area of application for the techniques is the roll up tables used in the management systems for database management systems to reduce the size of historic information about events that have occurred in the database management system. Each roll up entry in a roll up table is an aggregated entry that contains information about some number of events. A roll up entry that uses the techniques contains a representation of a set whose values are the occurrence times of the events that are represented by the rollup record. Among the techniques that can be used to represent the set of occurrence times are a comma list of the occurrence times and a bit map which has a bit for each second in a day. Roll up entries that contain such representations of sets of occurrence times may be analyzed to determine whether occurrences of events are related, and if they are, the fact of the relationship can be used to design filters that can be applied in the roll up process, in error reporting, and in the analysis of the roll up tables.