Compressed Data Structure for Ad-Hoc Analysis Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for ad-hoc data analysis require substantial hardware resources due to the need for large storage systems and vast computing power to handle and process extensive datasets, leading to high storage and computational demands.
Innovation Solution
A system that utilizes a compressed data structure comprising an aggregated dataset and pre-calculated metrics, allowing for on-demand analysis by storing pre-aggregated data and pre-computed statistics instead of raw data, significantly reducing storage and computational requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If raw data is stored for ad-hoc analysis, then analysis completeness is maintained, but storage space and computational resources are excessively consumed
Solution Approach 1:
The system pre-aggregates data and pre-calculates metrics before ad-hoc analysis is needed. The aggregated dataset stores summary statistics grouped by dimension variables, and the calculated dataset stores pre-computed metrics for each record. This preliminary processing reduces the amount of data that needs to be stored and processed during ad-hoc analysis, directly addressing the storage space consumption issue while maintaining analysis capability.
Solution Approach 2:
The system extracts only the necessary aggregated observations and pre-calculated metrics from the raw data for storage and analysis. Instead of storing all raw data, the system extracts and stores only the aggregated dataset and calculated dataset, which contain the essential information needed for ad-hoc analysis. This extraction significantly reduces storage requirements while preserving the ability to perform analyses.
2Quantity of substance
If large storage systems are deployed for raw data, then data volume is accommodated, but hardware costs and resource consumption increase
Solution Approach 1:
The system changes the state of data from raw form to aggregated and pre-calculated forms. By transforming the data representation from detailed raw records to summarized aggregated observations and pre-computed metrics, the system reduces the volume of data that must be stored and processed. This parameter transformation allows the same data to be represented in a more compact form, reducing hardware requirements.
3Productivity
If on-demand ad-hoc analysis is performed on raw data, then analysis flexibility is maintained, but computational time and processing power are excessively consumed
Solution Approach 1:
The system performs aggregation and metric calculation in advance before ad-hoc analysis is needed. The aggregated dataset contains pre-grouped observations by dimension variables, and the calculated dataset contains pre-computed metrics. When ad-hoc analysis is performed, the system can quickly query these pre-processed structures rather than processing raw data from scratch, significantly improving analysis speed and reducing computational power requirements.
4Quantity of substance
If compressed data structure with pre-aggregated data is used, then storage space is reduced, but data accessibility and analysis capability are limited
Solution Approach 1:
The compressed data structure serves multiple functions: the aggregated dataset provides pre-grouped data for various analysis queries, the calculated dataset provides pre-computed metrics, and together they enable a wide range of ad-hoc analyses. The structure is designed to be universally applicable for different analysis scenarios, maintaining versatility while achieving compression. The dimension variables and metrics are structured to support flexible querying and analysis.
Data Source
AI summary
A compressed data structure is disclosed for storing collected data and delivering ad-hoc data analysis. The compressed data structure can reduce hardware requirements, such as data storage requirements, and decrease processing requirements during ad-hoc data analysis. Raw data can be pre-aggregated to a base level of aggregation (base cell level) that is determined by the unique combination of attributes available for aggregation. Rather than storing the raw data, the entire distribution is stored at the base cell level along with a selection of pre-computed statistics of the raw data. At the time of ad-hoc analysis, pre-computed results based on the raw data can be retrieved or computed from the pre-computed statistics, or analysis of higher-level data can be computed on-demand based on the pre-aggregated base level data.