Distributed Dwarf Cube MapReduce Cardinality Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multidimensional Online Analytical Processing (MOLAP) systems face inefficiencies in handling large data sets due to unpredictable response times in Relational Online Analytical Processing (ROLAP) and the need for optimized storage and indexing in Multidimensional Online Analytical Processing (MOLAP), particularly in building and querying OLAP cubes.
Innovation Solution
A method and system for building a distributed dwarf cube using a MapReduce technique, which involves transforming data, generating indexes, sorting by cardinality, and partitioning data into blocks to create multiple dwarf cuboids that can be incrementally updated, allowing for efficient data processing and querying across multiple computers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is stored in relational databases (ROLAP), then data storage is flexible and easy to implement, but query response time becomes unpredictable and long for large data sets
Solution Approach 1:
The patent pre-computes and pre-stores aggregated data in multidimensional OLAP cubes before queries are executed. This preliminary action transforms raw transactional data into pre-aggregated cube structures, eliminating the need for time-consuming query computations on large relational datasets and ensuring predictable fast response times.
Solution Approach 2:
The patent transitions from traditional relational (flat) data storage to multidimensional OLAP cube structures with multiple hierarchical dimensions. This dimensional transformation enables efficient data aggregation and retrieval along different dimension hierarchies, providing both fast query performance and flexible analytical capabilities.
2Loss of time
If data is pre-computed and stored in OLAP cubes (MOLAP), then query response time is fast and predictable, but storage requirements increase and system complexity increases
Solution Approach 1:
The patent segments the OLAP cube into multiple dimension tables and fact tables organized in a star schema. This segmentation allows independent management of dimensions and facts, reducing overall system complexity while maintaining fast query performance through optimized joins between segmented components.
Solution Approach 2:
The patent introduces a cube builder component as an intermediary that automatically generates and maintains OLAP cube structures from relational data. This intermediary handles the complexity of cube construction, aggregation, and optimization, shielding the query system from complexity while delivering fast predictable response times.
3Adaptability or versatility
If OLAP cubes are built on large data sets, then comprehensive analysis is enabled, but processing time and computational resources increase significantly
Solution Approach 1:
The patent implements incremental cube building that processes and aggregates only the necessary portions of data required for specific analytical queries, rather than computing entire cubes on all available data. This partial action approach maintains comprehensive analysis capability while significantly reducing processing time and resource consumption.
Solution Approach 2:
The patent performs preliminary data filtering, aggregation, and transformation before cube construction, preparing only the relevant data subsets needed for specific analytical scenarios. This preliminary processing reduces the volume of data requiring intensive computation, enabling comprehensive analysis on large datasets with improved processing speed.
Data Source
AI summary
Systems and methods for building a distributed dwarf cube comprising dwarf cuboid using mapreduce technique are disclosed. Data comprising cube values and a cube definition may be received. The cube definition comprises dimensions defined for the cube values. The data received is processed. The data may be transformed to a format. Based upon the format of the data, indexes may be generated. The cube values in one or more dimensions may be sorted based on a cardinality of the cube values. The cube values may be sorted in an order of highest cardinality to lowest cardinality. The cardinality indicates distinctiveness of the cube values in the one or more dimensions. The data may be partitioned into data blocks. A dwarf cuboid may be built for one or more data blocks based upon the order of the cardinality of the cube values.