Partitioned Dictionary Memory Optimization in Business Intelligence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business intelligence systems face inefficiencies in memory optimization and data processing, particularly when handling large datasets with high cardinality, leading to issues like dictionary generation time-outs and memory overload due to the need to load large dictionaries across multiple nodes.
Innovation Solution
The implementation of an interest-driven business intelligence server system that generates partitioned dictionaries, where each partition contains a threshold number of values, and combines partitions with fewer entries to create larger ones, allowing for efficient data manipulation and storage in a common memory, and uses mapping and reducer processes to manage and reconcile dictionary identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large dictionaries are generated and loaded across multiple nodes for high cardinality data, then data processing completeness is improved, but memory usage and processing time increase significantly
Solution Approach 1:
The patent divides large dictionaries into smaller partitions based on data ranges. Each partition contains a subset of dictionary entries organized by value ranges, allowing nodes to process only relevant partitions rather than loading entire large dictionaries into memory. This segmentation reduces memory overhead and accelerates dictionary generation while maintaining complete data processing capability.
2Reliability
If large dictionaries are loaded across multiple nodes, then data coverage is improved, but memory usage on individual nodes increases
Solution Approach 1:
Dictionaries are partitioned into multiple smaller segments distributed across nodes. Each node stores only the partitions relevant to its data range, reducing individual node memory requirements while collectively maintaining complete data coverage across the distributed system.
Solution Approach 2:
The patent introduces a partition dimension to the dictionary storage model. Instead of a single flat dictionary structure, dictionaries are organized with an additional partition level, allowing data to be distributed across multiple dimensions (partition ID, value range, node location), thereby reducing the memory burden on any single node.
3Quantity of substance
If partitioned dictionaries are used to reduce memory load, then memory efficiency is improved, but system complexity increases
Solution Approach 1:
The system performs preliminary partitioning of dictionaries during initialization, organizing data into range-based partitions with associated metadata. This upfront preparation creates a structured framework that simplifies subsequent query operations, as nodes can directly access relevant partitions without complex runtime decision-making.
Solution Approach 2:
The patent introduces partition metadata as an intermediary layer between the distributed data and query operations. This metadata structure acts as a mediator that translates query requests into specific partition accesses, reducing the complexity of direct node-to-node coordination while maintaining memory efficiency.
Data Source
AI summary
Systems and methods for optimizing memory in an interest-driven business intelligence system in accordance with embodiments of the invention are illustrated. A dictionary for storing values of a dataset may be partitioned in accordance with some embodiments. The partitions of the dictionary may be generated by mapping and reducer processes. The mapping processes receive a value, determine the dimension of the data to which the value belongs, and provides the value to a reducer process that handles the determined dimension. Each reducer process generates partitions of the dictionary for each dimension. The number of values in each partition is determined and compared to a threshold value. Partitions that have a number of values greater than the threshold are stored in a common memory. Partitions smaller than the threshold value can be combined with other partitions such that the cardinality of the combined partition exceeds the threshold value.