Compressed Cube Index for Sparse OLAP Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data processing methods for sparse data in OLAP systems require excessive processing resources, leading to inefficiencies and high costs, especially when dealing with large and extremely sparse data sets.
Innovation Solution
The implementation of a compressed cube index that stores each distinct data value only once, along with information about its applicable ancestor tuples, allowing for fast aggregation and lean storage by using a method to generate and search an index defined by hierarchies, which reduces redundant storage and computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional aggregation methods are used on sparse data, then complete aggregation results are obtained, but processing time and computational resources become excessively large
Solution Approach 1:
The patent extracts only the essential information needed for aggregation by representing sparse data using a compressed cube index that stores only non-null values and their positions. This extraction eliminates redundant null entries while preserving aggregation completeness, enabling fast processing by working only with meaningful data points rather than entire dense arrays.
Solution Approach 2:
The patent transforms the data representation from a traditional two-dimensional table format to a multi-dimensional compressed cube structure. This dimensional transformation allows efficient aggregation by navigating through hierarchy levels (time, region, product categories) in a structured manner, reducing the computational space required while maintaining complete aggregation results.
2Speed
If traditional multidimensional array storage is used, then fast cell retrieval is achieved, but storage space becomes excessively large due to many null values
Solution Approach 1:
The patent removes null values from the storage structure by using a compressed cube index that only stores non-null data points along with their positional information. This extraction maintains the ability to retrieve cells efficiently through the index structure while dramatically reducing storage space by eliminating the need to store empty/ null entries throughout the multidimensional array.
Solution Approach 2:
The patent creates a compressed representation (copy) of the sparse data structure that preserves the essential information needed for retrieval and aggregation operations. This compressed copy uses a more efficient data structure that maintains query performance while using significantly less storage space compared to the original dense array representation.
3Volume of stationary object
If relational database row storage is used, then storage efficiency improves for sparse data, but aggregation performance becomes slower due to explicit dimension value storage
Solution Approach 1:
The patent transitions from relational database row-based storage to a multi-dimensional compressed cube structure. This dimensional change allows efficient aggregation by directly navigating through hierarchy levels without the overhead of explicit dimension value storage in each row. The compressed cube maintains storage efficiency for sparse data while enabling faster aggregation operations through its structured hierarchical organization.
Solution Approach 2:
The patent segments the data into hierarchical levels (dimensions) within the compressed cube structure. This segmentation allows aggregation operations to proceed level by level, processing only the relevant segments of data needed for each query rather than scanning entire datasets. The segmentation maintains storage efficiency while dramatically improving aggregation performance through targeted processing of data segments.
Data Source
AI summary
A method and data structure is described that builds summary information using processor time that is usually proportional to the size of input data and a depth of hierarchies for a plurality of attributes. The output of computation is stored in a smaller area by eliminating redundant storage and computation. An index is generated which includes tuples or rows that include lower bound values for each of the attributes, values of children of the lower bound values based on the hierarchies of the attributes, and coverage class indicators of the coverage classes of the children.