Density Metadata for Sparse Multi-Dimensional Data Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Sparse multi-dimensional data sources, commonly encountered in OLAP systems, lead to inefficient data retrieval and manipulation due to the need to visit numerous non-populated cells, resulting in poor performance.
Innovation Solution
A computer-readable storage medium with executable instructions to build population sets for dimensions of a multi-dimensional data source and identify intersections to form density metadata, which is accessed by a query processor to efficiently process queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional OLAP query processing is used on sparse multi-dimensional data, then the system can handle multi-dimensional analysis, but query processing efficiency deteriorates due to visiting numerous non-populated cells
Solution Approach 1:
The system performs preliminary action by pre-calculating and storing density metadata that describes the distribution of populated cells across dimensions before query execution. This metadata is generated in advance and stored in a structure that can be quickly accessed during query processing, eliminating the need to scan the entire multi-dimensional data structure.
Solution Approach 2:
Density metadata acts as an intermediary between the query processor and the sparse multi-dimensional data. Instead of directly accessing the sparse cube and visiting all cells, the query processor uses density metadata as a mediator to understand data distribution patterns, thereby guiding efficient access only to populated regions without needing to examine null values.
2Device complexity
If multi-dimensional data is stored in a dense format to simplify access, then data structure simplicity improves, but data storage requirements increase geometrically with the number of dimensions
Solution Approach 1:
The system segments the multi-dimensional data access problem by separating the dense data structure from the query processing logic. The density metadata is segmented into dimensional components that can be independently processed and combined, allowing the system to handle high-dimensional data without storing all possible combinations.
Solution Approach 2:
The invention introduces a new dimensional aspect by adding density metadata as an additional layer of information that describes the existing data structure. This metadata dimension contains information about cell population patterns, enabling the system to navigate sparse data efficiently without reducing the fundamental multi-dimensional storage requirements.
Data Source
AI summary
A computer readable storage medium includes a data analyzer to build population sets for dimensions of a multi-dimensional data source and to identify intersections of population sets so as to form density metadata. A query processor accesses the density metadata while processing a query.