Disk-Based Hash Map Merge for Memory-Efficient Query Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintaining and using multiple hash maps for large data sets consumes excessive storage space and processing capacity, reducing speed and throughput.
Innovation Solution
A disk-based merge system for hash maps that efficiently generates, merges, and combines hash maps using disjunctions and index values, allowing for memory-efficient lookup operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple hash maps are created to store large data sets, then storage capacity is increased, but memory usage and processing overhead increase, reducing speed and throughput
Solution Approach 1:
The patent divides the hash map into multiple disjunctions ( partitions), where each disjunction stores a subset of the data. This segmentation allows the system to handle large data sets by splitting them into manageable chunks that can be stored and processed more efficiently, reducing the overhead of maintaining a single large hash map while preserving storage capacity.
Solution Approach 2:
The patent introduces a new dimension of storage by writing hash map data to disk in addition to memory. This disk-based storage layer provides an alternative dimension for holding large data sets, allowing the system to scale storage capacity without proportionally increasing memory usage and processing overhead, thereby maintaining query speed.
2Quantity of substance
If multiple hash maps are maintained for large data sets, then storage space increases, but processing capacity is consumed
Solution Approach 1:
The patent extracts data from the active memory hash map and stores it on disk in disjunctions. This extraction removes the burden of maintaining large data structures in memory, reducing the processing capacity required while preserving the storage space needed for large data sets. The disk-based storage serves as an external repository that doesn't consume the same processing resources as in-memory structures.
3Quantity of substance
If multiple hash maps are used to store large data, then storage capacity is improved, but throughput is reduced
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing hash values and disjunction identifiers for each data element before actual query operations. This preliminary organization of data into disk-based disjunctions with pre-calculated indices allows for faster query processing, improving throughput by avoiding computationally expensive operations during actual queries while maintaining the storage capacity needed for large data sets.
Data Source
AI summary
Various embodiments for a disk-based merge for hash maps are described herein. An embodiment operates by identifying a plurality of hash maps with a plurality of disjunctions. The hash values of each of the entries may be moved to memory and compared for a particular disjunction. A data value with a lower hash value as determined based on the comparison is selected and stored in a merged hash map. The process is repeated until all the data values have been compared. A query is received, and processed based on the merged hash map.