Misaligned IO Sequence Deduplication via Rolling Offset Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques in storage arrays are inefficient due to their inability to handle misaligned data tracks, leading to resource-intensive CPU cycles and storage resource wastage, as they require generating multiple fingerprints to match shifted or misaligned data sets.
Innovation Solution
The proposed solution involves identifying sequences of input/output operations using an IO rolling offsets empirical distribution model, generating unique fingerprints for each data track, and updating a searchable fingerprint data structure to associate temporal parameters with IO operations, allowing for efficient deduplication of both aligned and misaligned sequences by reducing the need for extensive fingerprint generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional data deduplication techniques generate multiple fingerprints to match shifted or misaligned data sets, then matching accuracy is improved, but CPU cycles and storage resources are excessively consumed
Solution Approach 1:
The system performs preliminary actions by building an empirical distribution model of IO rolling offsets beforehand. This model captures the statistical characteristics of offset patterns in advance, allowing the deduplication process to directly compare new IO sequences against pre-computed offset distributions rather than generating multiple fingerprints through trial and error, thus reducing CPU consumption while maintaining matching accuracy
Solution Approach 2:
The invention changes the approach from generating multiple fingerprints with different offsets to using a single fingerprint comparison against an empirical distribution model. By transforming the problem from multiple discrete fingerprint generations to a continuous statistical model comparison, the system reduces computational overhead while preserving the ability to detect misaligned sequences
2Reliability
If traditional data deduplication techniques generate multiple fingerprints to handle misaligned data tracks, then deduplication completeness is improved, but storage resource utilization deteriorates
Solution Approach 1:
The system pre-computes and stores the empirical distribution model of IO rolling offsets, which encapsulates the patterns of misalignment. This preliminary action allows the system to handle misaligned data tracks efficiently during deduplication by comparing against the pre-built model rather than generating and storing multiple candidate fingerprints, thus maintaining deduplication completeness while reducing storage resource consumption
Solution Approach 2:
Instead of creating and storing multiple fingerprint copies for misaligned data tracks, the system uses a single fingerprint comparison against the empirical distribution model. The model acts as a virtual reference that captures the essence of multiple possible alignments without requiring physical storage of multiple fingerprint copies, thereby reducing storage resource usage while maintaining deduplication effectiveness
3Productivity
If the system uses an IO rolling offsets empirical distribution model to identify matching sequences, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The empirical distribution model serves as an intermediary between the raw IO sequences and the deduplication decision-making process. Rather than directly comparing IO sequences and making complex alignment decisions, the system uses the pre-computed statistical model as a mediator that encapsulates offset patterns, simplifying the comparison process while maintaining high processing efficiency
Solution Approach 2:
The invention transforms the complex problem of misaligned sequence matching into a parameter-based statistical comparison. By changing from discrete fingerprint generation to continuous statistical model comparison, the system achieves better processing efficiency. The increased complexity is offset by the fact that the empirical distribution model is built once and reused, making the per-IO processing simpler and faster
Data Source
AI summary
Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. In embodiments, a sequence of input/output (IO) operations in an IO stream received from one or more host devices by a storage array are identified. Additionally, a determination can be made as to whether a set of previously received IO operations match the identified IO sequence based on a time series relationship between the identified IO sequence and the previously received IO operations. Further, one or more data deduplication (dedup) techniques can be performed on the matching IO sequence.