Multidimensional Database Sparsity Removal via Filter Set Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business intelligence data queries in multidimensional databases face inefficiencies due to sparsity issues, leading to slow query processing and poor data presentation, as existing methods either fail to selectively remove null or zero data effectively or require extensive user interaction and knowledge.
Innovation Solution
A method and system for optimizing business intelligence data delivery by establishing a cross-tab with sparse data, selecting members for sparsity removal, generating a filter set expression based on collected contexts, and submitting queries to remove irrelevant, null, or zero data, thereby enhancing data relevance and query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If brute-force zero-suppression is used to remove all cells with null or zero value individually, then the implementation is simple and easy to execute, but the performance deteriorates significantly for multi-dimensional data sources
Solution Approach 1:
The patent segments the sparsity removal process into two phases: pre-processing phase where the sparse data matrix is identified and structured, and query-processing phase where the pre-computed sparsity information is utilized. This segmentation allows the system to avoid brute-force individual cell processing during queries, thereby improving performance while maintaining implementation simplicity through the use of structured sparsity representations.
Solution Approach 2:
The patent applies preliminary action by pre-processing the data to identify and structure sparse cells before actual queries are executed. The system pre-computes and stores sparsity information in a structured format that can be quickly applied during query processing, eliminating the need for expensive brute-force operations during query execution and significantly improving query processing performance.
2Productivity
If filtering member sets is used based on data value criteria, then sparsity can be removed selectively, but the usability deteriorates due to requiring in-depth knowledge and intensive user-interaction
Solution Approach 1:
The patent implements self-service by enabling the system to automatically detect, structure, and handle sparsity in multidimensional data without requiring user intervention. The system autonomously identifies sparse cells, pre-processes them into a structured representation, and applies appropriate filtering during queries based on predefined criteria, thereby maintaining high filtering effectiveness while significantly improving usability by eliminating the need for in-depth user knowledge and intensive interaction.
3Loss of time
If sparsity removal is performed in a single step using filter set expressions, then query processing time is reduced, but the complexity of the query system increases
Solution Approach 1:
The patent segments the sparsity removal functionality into a dedicated pre-processing module that operates independently from the query processing core. This modular segmentation allows the system to reduce query processing time by handling sparsity in advance while containing complexity within isolated pre-processing components, preventing the overall system complexity from becoming unmanageable.
Solution Approach 2:
The patent introduces an intermediary layer in the form of a sparsity representation structure that mediates between the raw multidimensional data and the query processing system. This intermediary pre-structures sparsity information in a format that is easy to consume during queries, reducing query processing time while shielding the query system from the underlying data complexity, thus balancing performance improvement with system complexity management.
Data Source
AI summary
Method and system for removing sparse data in a response to a query to a multidimensional database by establishing a cross-tab with sparse data. After selecting a member in an axis of the cross-tab for sparsity removal, a first set expression on the orthogonal axis is selected in the context of the member. The context of the orthogonal axis is collected for generating a filter set expression with a data removal criteria, based on the collected context; and the query is submitted to the database with the filter set expression.