Rollup Index Data Compression for Storage Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in efficiently managing and storing large volumes of raw historical data, leading to significant storage overhead and limitations in data analysis and visualization, as they often require decades of fine-grained data, which is costly and unnecessary.
Innovation Solution
The method involves generating rollup indices that summarize raw historical data using aggregation trees, allowing for reduced storage space by aggregating data into coarser time intervals and enabling efficient querying, while maintaining the ability to analyze and visualize both raw and rolled-up data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If raw historical data is stored in fine-grained detail for decades, then data analysis completeness is improved, but storage overhead increases significantly
Solution Approach 1:
The patent segments data storage into two distinct layers: raw data storage for recent periods and rollup index storage for historical periods. This segmentation allows the system to maintain complete raw data for detailed analysis when needed, while using compressed rollup indices for historical data to reduce storage overhead. The search service automatically determines which layer to query based on the time range and granularity requirements of the search parameters.
Solution Approach 2:
The patent changes the granularity parameter of data storage over time. Raw data is stored at fine granularity (e.g., per-minute intervals) for recent periods, while historical data is stored at coarser granularity through rollup indices (e.g., per-hour, per-day, or per-week intervals). This parameter change allows the system to optimize storage efficiency for historical data while preserving detailed information for recent data that requires fine-grained analysis.
2Measurement precision
If raw data is stored at fine time intervals, then query accuracy is improved, but storage space requirements increase
Solution Approach 1:
The patent implements a dynamic storage strategy where the granularity of data storage adapts based on the age of the data. Recent data is stored at fine time intervals to support accurate queries, while historical data is automatically rolled up to coarser intervals. The search service dynamically selects between raw data and rollup indices based on the query time range, ensuring query accuracy is maintained for recent data while optimizing storage for historical data.
3Adaptability or versatility
If all raw historical data is retained for visualization, then data visualization flexibility is improved, but storage costs increase
Solution Approach 1:
The patent introduces rollup indices as an intermediary layer between raw data and visualization queries. These indices provide summarized data at various time intervals, enabling flexible visualization of historical trends without requiring access to all fine-grained raw data. The search service acts as a mediator that automatically selects between raw data and rollup indices based on visualization requirements, maintaining flexibility while reducing storage costs for historical data.
Data Source
AI summary
Systems and methods for reducing data storage overhead are disclosed herein. In some embodiments, a system includes a rollup service that converts a raw data set into a rolled up index that takes up less storage than the raw data but is created in such a way that the rolled up index can be queried so as to generate responses that will substantially correspond to responses that would be generated using the raw data.