Partitioned Data Reduction for Memory-Efficient Hash Table Resizing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data reduction methods in relational database management and cloud computing face challenges with hash table resizing, resource estimation, and memory usage, particularly when dealing with large datasets and resource constraints.
Innovation Solution
A method for partitioned data reduction that forms groups of records based on common characteristics, performs intra-group and inter-group data reduction, and allocates resources efficiently through aggregation and spilling to secondary storage, ensuring effective resource management and reduced memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the hash table size is increased to avoid resizing, then resizing expense is reduced, but memory usage increases occupying memory needed for other operations
Solution Approach 1:
The hash table size is made dynamic rather than static. The system continuously monitors the fill factor and automatically resizes the hash table when the fill factor exceeds the threshold. This dynamic adjustment allows the system to adapt to changing data volumes, reducing resizing expense by only resizing when necessary while avoiding permanent over-allocation of memory.
2Quantity of substance
If the hash table size is decreased to save memory, then memory availability for other operations increases, but resizing expense increases when the table needs to be enlarged
Solution Approach 1:
The system performs preliminary sizing by estimating the final data volume using sampling techniques before fully allocating hash table resources. By predicting the required hash table size in advance, the system can allocate memory more accurately, reducing the need for subsequent enlargements and associated resizing expenses while still maintaining memory efficiency.
3Quantity of substance
If hash table resizing is performed frequently, then memory utilization is optimized, but synchronization complexity and processing overhead increase
Solution Approach 1:
The resizing operation is merged with the natural data insertion process. When the fill factor threshold is exceeded, the system performs a single coordinated resize operation that handles both the expansion and rehashing of existing elements in one unified process. This approach optimizes memory utilization while minimizing synchronization complexity by avoiding multiple separate resize operations.
Data Source
AI summary
Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.