Storage Page XOR Referencing for Low-Overhead Capacity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern log-structured storage systems face challenges in sub-sector deduplication due to high metadata representation and CPU cycle requirements, as well as reduced need for sub-sector dedupe after data compression, which limits capacity reduction efficiency.
Innovation Solution
The method involves performing a bit-wise exclusive OR (XOR) operation on sectors of candidate and target pages to identify identical data, storing XOR results, and determining entropy to decide whether to build a reference page for compression, thereby reducing storage capacity effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sub-sector deduplication is implemented to reduce storage capacity, then storage efficiency is improved, but metadata representation overhead and CPU cycle requirements increase
Solution Approach 1:
The patent divides pages into sectors and further into sub-sectors, enabling deduplication at the sub-sector level. This segmentation allows the system to identify and eliminate duplicates at a granular level without requiring metadata for entire pages, thereby reducing metadata overhead while improving storage efficiency.
Solution Approach 2:
The patent extracts only the necessary portions of data (identical sub-sectors) for deduplication processing. By using XOR operations to identify identical sub-sectors between pages and storing only the differences, the system reduces metadata requirements while achieving effective capacity reduction.
2Quantity of substance
If sub-sector deduplication is implemented to reduce storage capacity, then storage efficiency is improved, but CPU cycle requirements increase
Solution Approach 1:
The patent performs preliminary hashing of sub-sectors to create signatures that can be quickly compared. This preliminary action allows the system to identify potential duplicates before performing full XOR operations, significantly reducing CPU cycle requirements while maintaining effective deduplication.
Solution Approach 2:
The patent uses hash signatures as copies or representations of sub-sector data for comparison purposes. Instead of comparing entire sub-sectors directly, the system compares compact hash values, which requires minimal CPU cycles while still enabling accurate duplicate detection.
3Quantity of substance
If data compression is applied to reduce storage overhead, then storage efficiency is improved, but input/output access overhead increases
Solution Approach 1:
The patent applies compression selectively to specific data structures (such as metadata and deduplication indexes) rather than compressing all data uniformly. This local quality approach reduces storage overhead for critical components while minimizing the impact on I/O access performance for user data.
Data Source
AI summary
An aspect includes implementing capacity reduction in a storage system includes for each of a candidate page and a target page in the storage system, identifying a subset of sectors having identical data or a minimum amount of non-identical data, performing a bit-wise exclusive OR (XOR) operation on sectors of the candidate page and the target page, determining entropy from results of the XOR operation. Upon determining the entropy is less than or equal to a threshold value, an aspect includes building a reference page from an XOR sector containing results of the bit-wise XOR operation, and performing a compression operation on the reference page.