Probabilistic File Deduplication via Selective Chunk Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems face inefficiencies in processing unique files, as they require inserting all chunks into the chunk store, which wastes resources and does not result in storage savings, and lack effective methods to preserve locality and order of metadata, leading to slower data operations.
Innovation Solution
The solution involves determining whether a file is unique by selectively inserting only some or all of its chunks into the chunk store based on hash table presence, and organizing metadata to preserve locality, allowing for faster read and write operations by pre-fetching contiguous chunks into memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all chunks of a file are inserted into the chunk store for deduplication processing, then deduplication completeness is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent applies partial action by processing only a subset of chunks (e.g., first N chunks or chunks up to a size threshold) rather than all chunks. This allows the system to achieve reasonable deduplication results without the full time cost of processing every chunk, directly resolving the contradiction between completeness and processing time.
Solution Approach 2:
The patent segments the file processing into multiple phases: a quick phase that processes only a subset of chunks to determine potential duplicates, and a complete phase that processes all chunks only when needed. This segmentation allows the system to handle most files efficiently while maintaining reliability for important cases.
2Quantity of substance
If all chunks of a unique file are inserted into the chunk store, then storage capacity is utilized, but storage space is wasted without deduplication benefits
Solution Approach 1:
By processing only a subset of chunks from unique files, the system avoids inserting redundant data into the chunk store. This partial processing approach prevents storage space waste while still maintaining adequate storage capacity utilization for files that do contain duplicates.
Solution Approach 2:
The patent implicitly discards the opportunity to insert chunks from unique files into the chunk store after determining uniqueness through partial processing. This prevents wasting storage space on data that cannot be deduplicated, while recovering storage capacity for data that actually benefits from deduplication.
3Adaptability or versatility
If metadata is organized without preserving locality, then data access flexibility is improved, but read and write operation speeds decrease
Solution Approach 1:
The patent applies preliminary action by pre-organizing metadata with preserved locality and order information before data operations occur. This preliminary organization enables faster read and write operations through efficient data retrieval patterns while maintaining the flexibility needed for various access scenarios.
Solution Approach 2:
The patent adds a new dimension to metadata organization by incorporating locality and order information alongside traditional indexing. This multi-dimensional organization approach maintains data access flexibility while simultaneously improving operation speeds through optimized data retrieval paths.
Data Source
AI summary
Disclosed techniques include deduplication. Techniques include determining whether a file is unique, and depending on whether the file is unique, deduplicating only part of the file or the entire file. The techniques include processing the first chunk of a file to determine whether the hash of the chunk hash is already within a chunk hash table, and if not, then a percentage of chunks of the file is similarly processed. If any of the hashes of chunks are already in the chunk hash table, then at least some of file has been previously deduplicated, and file is not unique the storage system. If none of the processed chunks have a hash that is already in the chunk hash table, then the file is considered to be unique within chunk store and only a partial percentage of the file's chunks are deduplicated. Not all of a unique file's chunks are deduplicated.