Dynamic Data Structure Conversion for OLAP Cardinality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OLAP systems face inefficiencies in managing data cardinality across different ranges, requiring multiple data structures and consuming excessive memory by maintaining redundant representations, especially when transitioning between uncompressed lists, linear counters, and probabilistic counters.
Innovation Solution
Implementing a method that progressively converts data structures from an uncompressed list to a linear counter and then to a probabilistic counter (PCSA) without retaining the original data, reducing storage needs and extending the useful range of linear counters by eliminating redundant representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple data structures (uncompressed list, linear counter, probabilistic counter) are maintained simultaneously to handle different cardinality ranges, then accurate cardinality estimation is achieved across all ranges, but memory consumption increases significantly
Solution Approach 1:
The patent implements dynamic transitions between different data structure representations based on the current cardinality range. The system automatically converts from uncompressed list to linear counter, and then to probabilistic counter (PCSA) as cardinality increases, optimizing memory usage while maintaining accuracy for each range without maintaining all structures simultaneously
Solution Approach 2:
The patent changes the representation parameters of the data structure based on cardinality thresholds. For low cardinalities, it uses uncompressed lists; for medium cardinalities, it transitions to linear counters; for high cardinalities, it uses probabilistic counters. This parameter-based adaptation resolves the contradiction by matching the data structure to the appropriate cardinality range
2Measurement precision
If uncompressed lists are used to store data at low cardinalities, then exact cardinality information is maintained, but memory space is wasted when cardinality grows large
Solution Approach 1:
The system dynamically switches from uncompressed list representation to linear counter representation when cardinality exceeds a threshold. This dynamic transition ensures that exact cardinality information is maintained when the set is small, while memory space is conserved when the set grows large by using the more compact linear counter structure
3Quantity of substance
If linear counters are used to compress data at medium cardinalities, then memory usage is reduced, but accuracy is lost at very high cardinalities
Solution Approach 1:
The system dynamically transitions from linear counter representation to probabilistic counter (PCSA) representation when cardinality exceeds the linear counter's effective range. This dynamic adaptation allows the system to maintain memory efficiency while recovering accuracy for very high cardinalities through the probabilistic counter's ability to handle large ranges with bounded error
4Ease of operation
If redundant data structure representations are maintained for smooth transitions, then conversion between structures is facilitated, but memory footprint increases beyond database field limits
Solution Approach 1:
The patent extracts only the necessary cardinality information from the uncompressed list to initialize the linear counter, and then extracts only the cardinality estimate from the linear counter to initialize the probabilistic counter. This extraction approach eliminates the need to maintain redundant full representations, achieving smooth transitions while keeping memory footprint within database field limits
Solution Approach 2:
The system discards the uncompressed list representation once its cardinality information is extracted to initialize the linear counter. Similarly, it discards the linear counter once its cardinality estimate is extracted to initialize the probabilistic counter. This discarding and recovering approach ensures that only one representation exists at a time, maintaining ease of conversion while minimizing memory footprint
Data Source
AI summary
A cardinality of an incoming data stream is maintained in real time; the cardinality is maintained in a data structure that is represented by an unsorted list at low cardinalities, a linear counter at medium cardinalities, and a PCSA at high cardinalities. The conversion to the linear counter makes use of the data in the unsorted list, after which that data is discarded. The conversion to the PCSA uses only the data in the linear counter.