Byte-Level Single Instancing for Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional single instancing methods, such as file-based and block-based solutions, are inadequate in efficiently managing data storage by failing to effectively compare and match variable-sized data blocks, leading to inefficient storage and retrieval of duplicate data.
Innovation Solution
The implementation of byte-level or quasi byte-level single instancing using power of 2 rolling hashes and a search tree data structure allows for efficient comparison and storage of data blocks, enabling random access and compression, even when only a part of a block matches, thereby reducing storage needs and avoiding hash collisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If file-based single instancing is used, then storage space is reduced for identical files, but it fails to handle files with minor modifications efficiently
Solution Approach 1:
The patent divides files into fixed-size blocks and creates a block-level index structure. This segmentation allows the system to identify and single-instance only the identical blocks between files, while properly handling modified blocks. When a file is modified, only the changed blocks need to be re-stored, not the entire file, thus resolving the contradiction between storage efficiency and adaptability to modifications.
Solution Approach 2:
The patent applies partial single instancing by identifying and deduplicating only the matching portions (identical blocks) of files, rather than requiring complete file identity. This partial approach allows the system to achieve storage savings on common blocks while accommodating file modifications without losing the ability to handle versatile file changes.
2Quantity of substance
If block-level single instancing is used, then storage efficiency improves for files with minor changes, but inserting data at the start of files causes all subsequent blocks to shift and lose matching
Solution Approach 1:
The patent uses fixed-size blocks with a block-level index that independently tracks the position and content of each block. This segmentation approach means that when data is inserted or deleted, only the affected blocks and their index entries need to be updated, not the entire file structure. Blocks remain stable and can still match across files even if other blocks shift positions, resolving the contradiction between storage efficiency and structural stability.
Solution Approach 2:
The patent pre-establishes a block-level index structure that maps block contents to their positions independently of file structure. This preliminary indexing allows the system to quickly identify matching blocks regardless of their position in the file, and to efficiently update only the necessary blocks when modifications occur, maintaining both storage efficiency and structural stability.
3Device complexity
If conventional block-based hashing is used, then data comparison is simplified, but hash collisions increase and matching precision decreases
Solution Approach 1:
The patent divides data into fixed-size blocks and applies hashing at the block level rather than at the file level. This segmentation reduces the search space for hash comparisons and allows the use of simpler hash functions with fewer collisions. Each block is independently hashed and stored in an index, making comparison efficient while maintaining precision through block-level granularity.
Solution Approach 2:
The patent changes the parameter of hashing granularity from file-level to block-level. This parameter change allows the system to use more numerous, smaller hash values that are less prone to collisions. The block-level approach with fixed sizes creates a more distributed hash space, reducing collisions while keeping the comparison process simple through the block index structure.
Data Source
AI summary
Described in detail herein are systems and methods for deduplicating data using byte-level or quasi byte-level techniques. In some embodiments, a file is divided into multiple blocks. A block includes multiple bytes. Multiple rolling hashes of the file are generated. For each byte in the file, a searchable data structure is accessed to determine if the data structure already includes an entry matching a hash of a minimum sequence length. If so, this indicates that the corresponding bytes are already stored. If one or more bytes in the file are already stored, then the one or more bytes in the file are replaced with a reference to the already stored bytes. The systems and methods described herein may be used for file systems, databases, storing backup data, or any other use case where it may be useful to reduce the amount of data being stored.