Disk-Based Merge for Hash Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintaining and using multiple hash maps for large datasets consumes excessive storage space and processing capacity, reducing query speed and system throughput.
Innovation Solution
A disk-based merge approach for hash maps that involves sorting entries by hash value, assigning index values, and moving data values to disk locations, allowing for efficient merging and combination of hash maps without requiring all hash maps to be loaded into memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple hash maps are maintained for large datasets, then storage capacity and data organization are improved, but storage space consumption and processing capacity increase excessively
Solution Approach 1:
The patent segments the hash map data into multiple disjoint sets, where each set contains a portion of the hash map entries. This segmentation allows the system to process and store data in smaller, manageable units rather than maintaining one large hash map in memory, thereby reducing storage space consumption and processing overhead while still providing comprehensive data organization capability.
2Quantity of substance
If multiple hash maps are maintained for large datasets, then data organization is improved, but query speed and system throughput decrease
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing disjoint sets of hash map entries on disk before queries are executed. When a query arrives, the system can quickly retrieve the relevant pre-processed disjoint set from disk and perform the lookup, avoiding the need to scan through multiple full hash maps in memory during query execution, thus maintaining query speed.
3Productivity
If hash maps are merged into a single structure, then query speed is improved, but memory usage increases excessively
Solution Approach 1:
The patent introduces an intermediary mechanism using disjoint sets that act as intermediate structures between the raw hash map data and the final merged result. Instead of loading all hash map data into memory for merging, the system processes and merges disjoint sets incrementally, with each set being an intermediate representation that requires minimal memory to handle, thus enabling memory-efficient merging while still achieving the speed benefits of a merged structure.
Data Source
AI summary
Various embodiments for a disk-based merge for combining merged hash maps are described herein. An embodiment operates by identifying a first hash map and a second hash map, and comparing a first hash value from the first hash map with a second hash value from the second hash map, with the lowest index values. A lowest hash value is identified based on the comparison, and an entry corresponding to the lowest hash value is stored in a combined hash map. This process is repeated until all of the hash values from both the first set of hash values and the second set of hash values are stored in the combined hash map. A query is received, and processed based on the combined hash map.