Multidimensional Database Hierarchical Aggregation for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multidimensional database environments face challenges in efficiently processing large data sets due to the increasing size of data sources, making it difficult to persist data at the leaf node level, especially in big data environments, where loading all leaf nodes into memory is undesirable.
Innovation Solution
The system organizes data into a hierarchical structure with multiple levels of nodes, loads source data into leaf nodes, aggregates it to higher levels using metadata, and discards the source data from the leaf nodes to free memory, allowing for load, aggregate, and batch calculations in a single scan, reducing the data load by a factor of 10 to 100 times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If source data is loaded into all leaf nodes in a multidimensional database, then complete data availability is achieved, but memory requirements increase significantly
Solution Approach 1:
The patent segments the multidimensional database into a hierarchical structure with multiple levels of nodes (leaf nodes and parent nodes). Data is segmented and stored only at the leaf nodes, while parent nodes contain aggregation results. This segmentation allows the system to avoid loading all data into memory simultaneously, as only leaf node data and necessary aggregation results need to be retained.
Solution Approach 2:
The patent extracts aggregation results from the detailed source data at leaf nodes and stores them at parent nodes. This extraction process allows the system to discard or reduce the retention of detailed source data after aggregation, thereby reducing memory requirements while maintaining the ability to answer queries through the extracted aggregation results.
2Speed
If all leaf node data is persisted in memory, then fast access to detailed data is enabled, but processing efficiency decreases due to excessive data load
Solution Approach 1:
The hierarchical node structure segments data access into different levels. For detailed data access, the system navigates to specific leaf nodes rather than scanning all data in memory. For summary queries, the system accesses parent nodes directly. This segmentation improves both access speed and processing efficiency by avoiding unnecessary data loading and processing.
Solution Approach 2:
The system performs partial loading of data into memory, loading only the leaf node data and aggregation results that are currently needed, rather than loading all possible data. This partial action approach maintains fast access to required data while avoiding the processing overhead of managing excessive data loads.
3Loss of information
If hierarchical aggregation is implemented across multiple levels, then data summarization is improved, but system complexity increases
Solution Approach 1:
The patent segments the aggregation process into discrete hierarchical levels, where each level aggregates data from its child nodes. This segmentation makes the complex aggregation process manageable by breaking it into smaller, organized steps across multiple levels, reducing the apparent complexity while maintaining comprehensive data summarization.
Solution Approach 2:
The patent introduces a hierarchical dimension to the data structure, organizing nodes into multiple levels (leaf nodes at the bottom, parent nodes above). This dimensional organization simplifies the aggregation process by providing a clear upward hierarchy for data summarization, making the system more manageable despite the added structural dimension.
4Reliability
If source data is retained at leaf nodes after aggregation, then data integrity is maintained, but memory usage increases unnecessarily
Solution Approach 1:
The patent extracts aggregation results from the source data at leaf nodes and stores them at parent nodes. Once extraction is complete, the source data at leaf nodes can be discarded or not loaded into memory, as the extracted aggregation results preserve the necessary information for queries. This extraction process maintains data integrity for analytical purposes while reducing memory usage.
Solution Approach 2:
The system discards source data at leaf nodes after their aggregation values have been calculated and stored at parent nodes. The aggregation results are recovered and stored at the parent level, allowing the system to eliminate the need to retain detailed source data in memory while preserving the aggregated information needed for queries and analysis.
Data Source
AI summary
In accordance with an embodiment, systems and methods can provide for load, aggregate, and batch calculation in in a multidimensional database environment. Metadata can be used to identify in which higher dimensions each leaf-level load is used. At such time, the data from the leaf level is aggregated to each of the higher dimensions where it is to be used. Then, the aggregate values are stored in memory until a completed sum (or other calculation) is completed at the higher levels. At such time, the aggregate values can be discarded from memory. The ability of the system to perform a load, aggregate, and batch calculation in one scan is particularly useful in processing very large sets of data, for example in big data environments, where it may not be desirable to load the data for all of the leaf nodes into current memory.