Disk-Based Hash Map Merge for Faster Large-Scale Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Maintaining and using multiple hash maps for large data sets consumes excessive storage space and processing capacity, reducing query speed and throughput.
Innovation Solution
A disk-based merge system for hash maps that efficiently generates, merges, and combines hash maps using disjunctions, allowing parallel processing and minimizing memory usage by storing and ordering data values on disk before comparison and merging in memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple hash maps are created to store large data sets, then storage capacity is increased, but memory consumption and processing overhead increase, reducing query speed and throughput
Solution Approach 1:
The patent divides the large data set into multiple disjunctions, where each disjunction is stored in a separate hash map. This segmentation allows the system to manage large quantities of data while maintaining efficient query performance by processing only relevant disjunctions during lookups, rather than searching through all data in a single large hash map.
Solution Approach 2:
The patent introduces a hierarchical dimension to hash map organization by creating multiple levels of hash maps (primary and secondary). This dimensional approach allows efficient navigation through large data sets by first locating the relevant primary hash map and then accessing specific entries within it, reducing the search space for queries.
2Quantity of substance
If multiple hash maps are maintained for large data sets, then storage capacity is increased, but processing capacity is consumed
Solution Approach 1:
The patent implements partial action by loading only the necessary disjunctions into memory based on query patterns and data access requirements. Instead of maintaining all hash maps in memory simultaneously, the system loads subsets as needed, reducing memory consumption and processing overhead while still providing access to the complete data set when required.
3Adaptability or versatility
If multiple hash maps are used to store large data, then storage flexibility is improved, but memory usage increases
Solution Approach 1:
The patent implements dynamic memory management where hash maps and disjunctions are loaded into and evicted from memory based on actual query workloads and access patterns. This dynamic approach allows the system to adapt memory usage to current needs, maintaining storage flexibility for large data sets while optimizing memory consumption by keeping only actively accessed data in fast memory.
Data Source
AI summary
Various embodiments for a disk-based merge for hash maps are described herein. An embodiment operates by identifying a plurality of hash maps with a plurality of disjunctions. The hash values of each of the entries may be moved to memory and compared for a particular disjunction. A data value with a lower hash value as determined based on the comparison is selected and stored in a merged hash map. The process is repeated until all the data values have been compared. A query is received, and processed based on the merged hash map.