Selective Aggregation Paths for Memory-Efficient Data Cubes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data cube model generates redundant dimension combinations that cause unnecessary memory usage and performance overhead by combining all dimensions, even if they do not exist in the data source.
Innovation Solution
A method and apparatus for data aggregation that identifies and aggregates only relevant dimension combinations by obtaining an aggregation basis and calculation dimension from the data source, recording a correspondence relationship between target aggregation paths and results, thereby avoiding unnecessary memory usage and performance overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all dimensions are combined to form a complete data cube structure, then data query and retrieval efficiency is improved, but memory usage and performance overhead increase due to redundant dimension combinations
Solution Approach 1:
The patent extracts and stores only the aggregation paths that actually exist in the data source into a separate aggregation path mapping structure. This separates the necessary dimension combinations from the redundant ones, allowing the system to maintain query efficiency while reducing memory usage by storing only relevant paths rather than all possible dimension combinations.
Solution Approach 2:
The patent segments the dimension combinations into two categories: those that exist in the data source and those that do not. By creating a segmented view of the data cube structure, the system can optimize for both query performance (by having pre-computed paths for existing combinations) and memory efficiency (by not storing paths for non-existing combinations).
2Adaptability or versatility
If all dimension combinations are generated in the data cube, then comprehensive data analysis capability is improved, but performance overhead increases due to processing redundant aggregations
Solution Approach 1:
The patent extracts only the meaningful aggregation paths from the complete set of dimension combinations. By identifying and storing only those paths that correspond to actual data relationships, the system maintains comprehensive data analysis capability while eliminating the performance overhead associated with processing and storing redundant aggregation paths.
Solution Approach 2:
Instead of generating all possible dimension combinations (excessive action), the patent generates and stores only the necessary aggregation paths that exist in the data source (partial action). This selective approach provides sufficient data analysis capability for practical purposes while significantly improving processing efficiency by avoiding unnecessary computations.
3Loss of information
If redundant Cuboids are created to ensure complete dimension coverage, then data completeness is improved, but unnecessary memory usage occurs
Solution Approach 1:
The patent extracts and stores only the aggregation paths that represent actual data relationships in the source. This extraction approach ensures that all meaningful information is preserved (data completeness) while eliminating redundant Cuboids that would consume memory without adding value.
Solution Approach 2:
The patent discards redundant dimension combinations that do not exist in the data source. By selectively removing these unnecessary structures, the system recovers memory resources while maintaining completeness of the actual data relationships through the stored aggregation paths.
Data Source
AI summary
The application discloses a method, apparatus, electronic device and storage medium for data aggregation. The method includes: obtaining an aggregation basis dimension and an aggregation calculation dimension, wherein the aggregation basis dimension and the aggregation calculation dimension are respectively fields in a data source table to be aggregated; obtaining an aggregation result by aggregating field values corresponding to the aggregation calculation dimension of at least one target row, wherein field values of the aggregation basis dimension of the at least one target row are the same; and recording a correspondence relationship between a target aggregation path and the corresponding aggregation result, wherein the target aggregation path is determined according to field values of the aggregation basis dimension of the target row.