Storage Page XOR Deduplication for Low-Overhead Capacity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in sub-sector deduplication due to high metadata representation and CPU cycle requirements, as well as increased IO access overhead from data compression, which limits the effectiveness of capacity reduction techniques.
Innovation Solution
Implementing a sector-based deduplication method using an exclusive OR (XOR) operation to identify and store non-identical sectors, compressing the XOR results instead of unique data, thereby reducing storage capacity while minimizing metadata and CPU usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to reduce storage space, then storage efficiency is improved, but input/output access overhead increases
Solution Approach 1:
The patent segments data into fixed-size blocks and applies deduplication at the block level rather than compressing entire files. This segmentation allows the system to identify and eliminate duplicate blocks independently, reducing storage space without requiring full-file compression operations that increase I/O overhead.
Solution Approach 2:
The patent extracts and stores only the unique portions of data blocks after identifying duplicates through hashing. By taking out only the necessary unique data rather than compressing entire datasets, the system reduces storage requirements while minimizing the processing overhead associated with reading and writing compressed data.
2Quantity of substance
If sub-sector deduplication is implemented to reduce storage capacity, then storage efficiency is improved, but metadata representation and CPU cycle requirements increase
Solution Approach 1:
The patent divides data into sector-aligned blocks and processes them independently with fixed-size segmentation. This approach reduces metadata overhead compared to variable-length sub-sector deduplication, as each block has a predetermined size and requires standardized metadata structures, thereby lowering CPU cycle requirements for metadata management.
Solution Approach 2:
The patent changes the deduplication granularity parameter from sub-sector level to sector-level fixed blocks. This parameter change simplifies the metadata structure and reduces CPU overhead by eliminating the need to track variable-length sub-sector boundaries and their associated metadata, while still achieving effective duplicate elimination.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The XOR-based deduplication technique efficiently conserves storage capacity with minimal metadata and computational overhead, enhancing storage efficiency and reducing resource consumption.
Implementation Method 1
an exclusive OR (XOR) operation is performed on the two identical pages, and a result page is generated
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An aspect includes implementing capacity reduction in a storage system includes for each of a candidate page and a target page in the storage system, identifying a subset of sectors having identical data or a minimum amount of non-identical data, performing a bit-wise exclusive OR (XOR) operation on sectors of the candidate page and the target page, determining entropy from results of the XOR operation. Upon determining the entropy is less than or equal to a threshold value, an aspect includes building a reference page from an XOR sector containing results of the bit-wise XOR operation, and performing a compression operation on the reference page.