OLAP Cube Delta Ingestion for Fast Multidimensional Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating new multidimensional array data structures, such as OLAP cubes, is computationally costly when ingesting significant quantities of data, especially in enterprise resource planning systems.
Innovation Solution
A system that performs delta calculations to append new data to existing OLAP cubes using the SSAS 'Process Add' method, avoiding the need to regenerate the cubes during ingestion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If new multidimensional array data structures (OLAP cubes) are generated during data ingestion, then data is stored in a structured format suitable for analysis, but computational resources and time are excessively consumed
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing aggregate values in the OLAP cube before data ingestion occurs. When new data arrives, only delta (difference) values need to be computed and applied, rather than regenerating entire cubes. This preliminary preparation of the data structure enables efficient incremental updates.
Solution Approach 2:
The patent segments the data ingestion process into two distinct phases: (1) computing delta values by comparing new data with existing cube data, and (2) applying only the computed deltas to update the OLAP cube. This segmentation avoids the need to process and regenerate the entire cube structure, significantly reducing computational overhead.
2Loss of information
If existing OLAP cubes are rebuilt during each ingestion operation, then data currency is maintained, but system performance and response time deteriorate
Solution Approach 1:
The patent implements dynamics by transitioning from a static cube regeneration approach to a dynamic incremental update approach. The system continuously adapts the OLAP cube by applying only the necessary delta changes, allowing the cube to evolve efficiently as new data arrives without requiring complete rebuilds, thus maintaining data currency with minimal time loss.
Solution Approach 2:
The patent extracts only the essential delta (difference) information from the comparison between new incoming data and existing cube data. By taking out and processing only these differential values rather than the entire dataset, the system maintains data currency while dramatically reducing the time required for each ingestion operation.
3Measurement precision
If complete OLAP cubes are regenerated for every data update, then data accuracy is ensured, but computational cost increases significantly
Solution Approach 1:
The patent applies partial action by computing and applying only the necessary delta changes required to maintain data accuracy, rather than performing excessive full cube regeneration. This partial processing approach ensures data accuracy is maintained while significantly reducing computational energy consumption by avoiding redundant calculations of unchanged data portions.
Data Source
AI summary
A multidimensional data aggregation and ingestion system includes a multidimensional data aggregation and ingestion computing device that ingests data received from one or more sources. During operation, data is received from one or more internal or external data sources. Next, received data is aggregated in accordance with aggregation rules. Next, delta information is determined by comparing received data to stored data in accordance with delta determination rules. Next, delta information is appended to the stored data. Data is ingested without deleting stored data. A stored multidimensional array data structure is updated based on received data without generating a new stored multidimensional array data structure. For example, a “Process Add” method provided by SQL Server Analysis Services (SSAS) is used to append delta information to a stored Online Analytical Processing (OLAP) cube. The novel system ingests significant quantities of data without needing to generate new OLAP cubes on each ingestion operation.