Sorting Fingerprint Data with Limited Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication processes in network storage systems are inefficient and memory-intensive, leading to scalability issues when handling large data sets, as they require significant memory resources for sorting fingerprint files, which is not feasible in enterprise-scale systems.
Innovation Solution
The method involves creating temporary files from an input file, defining groups from these temporary files, and using multiple execution threads to sort the contents concurrently within a limited memory space, allowing for efficient sorting and merging of data without requiring extensive memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional sorting techniques are used to sort fingerprint files for deduplication, then the sorting can be performed, but the memory requirements become excessively large and the process becomes extremely time-consuming
Solution Approach 1:
The patent divides the fingerprint file into multiple smaller chunks that can be processed in parallel. Each chunk is sorted independently using limited memory, and then the sorted chunks are merged. This segmentation allows the system to handle large fingerprint files without requiring excessive memory, as each chunk fits within the available memory constraints while still enabling efficient processing through parallelization.
Solution Approach 2:
The patent introduces a temporal dimension to the sorting process by performing sorting operations across multiple time steps rather than attempting to sort the entire file in a single memory operation. The process involves multiple passes where fingerprints are sorted and merged iteratively, transforming the problem from a single-memory bottleneck to a multi-step process that utilizes available memory efficiently over time.
2Manufacturing precision
If the entire fingerprint file is loaded into memory for sorting, then sorting can be performed efficiently, but the system runs out of available memory and cannot process large data sets
Solution Approach 1:
The patent segments the fingerprint file into manageable chunks that individually fit within the available memory. Each chunk is processed independently, allowing efficient sorting within memory constraints. The segmentation strategy ensures that no single operation requires more memory than is available, while still maintaining sorting efficiency through optimized chunk processing and merging.
Solution Approach 2:
The patent performs preliminary sorting of individual chunks before the final merging process. By pre-sorting smaller chunks that fit in memory, the system prepares data in an organized manner that facilitates efficient merging. This preliminary action allows the system to achieve overall sorting efficiency without requiring the entire file to be loaded into memory at once.
3Quantity of substance
If fingerprint files are sorted separately and sequentially, then memory usage is minimized, but the total processing time becomes inordinately long
Solution Approach 1:
The patent merges multiple sorting operations into a unified process that handles multiple fingerprint chunks simultaneously. By combining the sorting and merging operations into an integrated workflow that processes multiple chunks in parallel, the system reduces total processing time while maintaining low memory usage. The merging strategy efficiently combines results from multiple chunks without requiring all chunks to be processed sequentially.
Solution Approach 2:
The patent adds a parallel processing dimension to the sorting workflow by executing multiple sorting and merging operations simultaneously across different time steps. This temporal parallelization allows the system to process multiple fingerprint chunks concurrently, dramatically reducing total processing time while each individual operation continues to use minimal memory. The multi-dimensional approach combines spatial segmentation with temporal parallelization for optimal performance.
4Productivity
If more memory is allocated for sorting operations, then sorting performance improves, but the space available for servicing user requests decreases
Solution Approach 1:
The patent segments the sorting workload into small, memory-efficient chunks that can be processed with minimal memory allocation. This segmentation allows the system to maintain small memory footprints during sorting operations, preserving memory capacity for user requests. The chunked approach enables sorting performance to improve through parallel processing while each individual chunk remains small enough to not impact user request handling capacity.
Solution Approach 2:
The patent implements dynamic memory management where the sorting process adapts its memory usage based on the available memory constraints. The system dynamically adjusts the size and number of processing chunks to fit within the memory available for both sorting and user requests. This dynamic adaptation ensures that sorting performance is optimized without compromising the system's ability to service user requests, as memory resources are flexibly allocated based on operational needs.
Data Source
AI summary
An efficient and highly scalable method of sorting an input file in a processing system by using only a limited amount (i.e., a portion) of memory in the processing system, where that amount of memory is substantially smaller than the input file, is disclosed. The input file can be, for example, a fingerprint database for use in deduplication, and the processing system can be, for example, a network storage server. The merge phase is broken down into sub-phases, where each sub-phase takes a predetermined number of subsets of a fingerprint file to merge and writes them back as a sorted, merged group. The number of threads used to process these groups can depend on the number of central processing units (CPUs) present in the system and can be dynamically tuned to achieve desired level of performance.