Adaptive Payload Management for Database Aggregation Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face inefficiencies in processing aggregations due to the high memory and processing costs associated with maintaining payload formats, particularly when dealing with large sets of distinct values, as the same payload format is not efficient for all aggregation groups and converting between formats is expensive.
Innovation Solution
A system and method for dynamically migrating the payload format based on characteristics such as the aggregation function and historical knowledge, using tools like a director and manager to assess and select optimal payload formats like lists, bitmaps, and hash tables, ensuring efficient memory and processing usage by adapting to changing resource availability and aggregation sizes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a hash table payload format is used to store distinct values, then the system can handle large sets of values, but memory consumption increases significantly
Solution Approach 1:
The patent implements dynamic payload format migration that adapts the data structure based on the actual number of distinct values encountered during aggregation. The system starts with a compact format and migrates to a hash table only when necessary, making the memory allocation dynamic rather than static, thus resolving the contradiction between handling large sets and minimizing memory consumption.
Solution Approach 2:
The system changes the parameter of payload format selection based on the characteristic of the aggregation (number of distinct values). By monitoring the cardinality of distinct values and adjusting the payload format accordingly, the system optimizes memory usage while maintaining the capability to handle both small and large sets of distinct values.
2Productivity
If the payload format is changed to optimize for different aggregation sizes, then processing efficiency improves, but conversion between formats increases processing cost
Solution Approach 1:
The patent employs a progressive migration strategy where the system prepares for format changes by monitoring aggregation characteristics in advance. The migration is triggered only when specific thresholds are met, avoiding unnecessary format conversions and their associated overhead, thus improving processing efficiency while minimizing format conversion time.
Solution Approach 2:
The system maintains continuous aggregation processing by implementing incremental format migration that allows the aggregation to continue without complete interruption. The migration is performed in a manner that preserves the aggregation state and minimizes downtime, ensuring continuity of useful action.
3Device complexity
If a single payload format is used for all aggregation groups, then device complexity is reduced, but processing efficiency decreases for minimal set aggregations
Solution Approach 1:
The patent segments the aggregation processing into different phases with different payload formats. By dividing the aggregation process into initial phase (using compact formats) and extended phase (using hash tables when needed), the system achieves both simplicity and efficiency by applying the appropriate format to each segment of the processing workflow.
Solution Approach 2:
The system implements a universal aggregation framework that can handle multiple payload formats through a unified interface. The aggregation logic remains the same while the underlying storage format adapts to the specific needs of each aggregation group, providing multi-functionality without increasing device complexity.
Data Source
AI summary
Embodiments relate to payload storage format for storing data in support of an aggregation function. As an input is subject to aggregation, the input is evaluated to ascertain a payload format for the aggregation. It is understood that there is more than one payload format. An evaluation of the aggregation key is a factor in the initial payload format. If the key is an addition to an existing aggregation, the evaluation considers changing the format of the payload to address processing and/or memory efficiency for the aggregation. The evaluation and the format change takes place dynamically so that the aggregation may continue.