Distributed OLAP with Associative Array Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OLAP systems face limitations in handling large amounts of data, numerous dimensions, and real-time updates, making them impractical for advanced applications like live dashboards and predictive analytics.
Innovation Solution
The system parses and enriches event streams by partitioning data into dimension and metric partitions, using a distributed associative array with a two-level caching mechanism and atomic updates, allowing for scalable storage and processing across multiple hosts, enabling efficient handling of increased data volumes and dimensions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If OLAP data is stored in RAM for real-time query response, then query response speed is improved, but the size of OLAP data is limited by RAM capacity
Solution Approach 1:
The patent divides the OLAP data storage into multiple segments: hot data stored in RAM for fast access and cold data stored in persistent storage for capacity. The system segments the data cube into different partitions that can be independently managed and loaded into memory as needed, allowing the system to handle data sizes far exceeding RAM capacity while maintaining fast query performance for frequently accessed data.
2Adaptability or versatility
If the number of dimensions and cardinality are increased to handle more complex queries, then query versatility is improved, but the data size and processing complexity increase exponentially
Solution Approach 1:
The patent introduces a new dimension to the traditional OLAP data model by adding a hierarchical dimension structure with multiple levels of aggregation. This allows the system to manage high cardinality dimensions by organizing data in hierarchical groups, reducing the effective complexity for queries while maintaining the ability to analyze at multiple granularities. The hierarchical dimension adds a new axis for data organization without increasing the exponential complexity of the base data cube.
3Speed
If pre-calculation is performed for likely queries to improve response time, then query response speed is improved, but the system cannot handle new or unexpected queries efficiently
Solution Approach 1:
The patent implements preliminary action by pre-calculating and storing aggregated data at multiple hierarchical levels and for frequently accessed dimension combinations. The system performs advance computation of data cube partitions and stores them in an optimized format in persistent storage. When queries arrive, the system can quickly assemble results from these pre-computed partitions, providing fast response for both anticipated and novel queries without requiring complete pre-calculation of all possible query results.
Data Source
AI summary
Source data of an event stream is parsed and supplemented with additional data from reference data sources, producing an enriched event stream from the parsed event stream data. The data records of the enriched event stream are partitioned into data fields designated as a dimension partition and a metric partition, which are partitioned into sub-dimension projections mapped to a plurality of storage keys, such that each of the storage keys includes one or more placeholder wildcard values and each of the storage keys is stored into a database of the computer system by the computer processor. The stored storage keys are then aggregated onto a two-dimensional coordinate vector such that, if the computer processor identifies a permuted storage key having metric values for which a corresponding storage key already exists in the database, then the computer processor aggregates the metric values of the identified storage key with the metric values of the corresponding storage key, and if the computer processor does not identify the permuted storage key as having a corresponding storage key that already exists in the database, then the computer processor writes the metric values of the permuted storage key into the database, comprising initial values for the key combination of dimension values.