Multi-dimensional Query Processing via Tree Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In shared nothing architecture distributed computing systems, storing and querying multi-dimensional data is challenging due to data explosion, computational intensity, and skewed partition distributions, leading to slow query performance and high storage costs.
Innovation Solution
The system instantiates computationally expensive multi-dimensional tuples, utilizes Expensive Dimension Level Materialization (EDLM), Hash Partitioning of Materialized Dimension Levels (HPMDL), and Orthogonal Dimension Level Partitioning (ODLP) to reduce computational load and storage complexity, while using Family Operation Resolution using Metadata (FORM) for efficient query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multi-dimensional data is stored in a shared nothing distributed system, then scalability is improved, but query performance deteriorates due to data explosion and computational intensity
Solution Approach 1:
The system performs preliminary aggregation of data at lower dimensions before queries are executed. By pre-computing and storing aggregated data for lower-dimensional tuples, the system avoids the need to scan and process entire multi-dimensional datasets during query time, thereby maintaining scalability while improving query performance.
Solution Approach 2:
The patent segments the multi-dimensional data into hierarchical levels (lower-dimensional and higher-dimensional tuples) and stores them separately. This segmentation allows the system to process only the necessary segments during queries rather than handling the complete dataset, reducing computational intensity while preserving the distributed scalability advantage.
2Productivity
If computationally expensive multi-dimensional tuples are processed, then query results are obtained, but storage costs increase due to data explosion
Solution Approach 1:
Instead of materializing all possible multi-dimensional tuples (excessive action), the system selectively materializes only the lower-dimensional tuples that are most frequently queried or have the highest aggregate values. This partial materialization approach reduces storage requirements while still providing efficient query results for the most common access patterns.
Solution Approach 2:
The system applies different storage strategies to different parts of the data hierarchy. Lower-dimensional tuples with high query frequency are materialized and stored locally at their dimension level, while higher-dimensional tuples are stored only when needed or in a compressed form. This local quality differentiation optimizes the balance between storage costs and query performance.
3Adaptability or versatility
If data is partitioned in a shared nothing architecture, then scalability is achieved, but partition skew creates slow query performance
Solution Approach 1:
The system performs preliminary aggregation at lower dimensions before data is distributed to partitions. By pre-computing aggregates at lower-dimensional levels, the system ensures more uniform data distribution across partitions, reducing partition skew and improving query processing speed while maintaining scalability.
Solution Approach 2:
The patent changes the partitioning parameter from individual row-level data to aggregated data at lower dimensions. This parameter change results in more balanced partition sizes and reduces skew, as aggregates at lower dimensions tend to be more uniformly distributed than individual records, thereby improving query performance in the shared nothing architecture.
Data Source
AI summary
A computer readable storage medium includes executable instructions to receive a request for data from a multi-dimensional cube divided into partitions distributed across nodes supporting a shared nothing distributed multi-dimensional database. The request is transformed into physical access layer operators in a tree structure. The tree structure is reduced into an index scan operator.