Deduplication Boundary Markers for Data Stream Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data integrity verification data, such as checksums, can degrade deduplication efficiency in storage systems and complicate data recovery, especially during backup and migration across different storage tiers, due to lack of awareness of file boundaries in storage systems.
Innovation Solution
Embedding checksum markers and boundary markers within data streams to enable storage systems to recognize and process data integrity information separately from the actual data, allowing for optimized deduplication and improved data access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If integrity verification data (checksums) are included in the data stream, then data integrity can be verified during restoration, but deduplication efficiency is seriously degraded
Solution Approach 1:
The patent segments the data stream into distinct regions: checksum marker region, data region, and boundary markers. This segmentation allows the storage system to process checksums and data separately, maintaining integrity verification while improving deduplication efficiency by avoiding checksum data from degrading the deduplication process.
Solution Approach 2:
The patent extracts checksum data from the data stream by introducing dedicated checksum markers that delimit checksum regions. This extraction allows the storage system to separate integrity verification functions from the actual data deduplication process, resolving the conflict between maintaining integrity and achieving efficient deduplication.
2Productivity
If storage systems are unaware of file boundaries, then data can be processed as continuous streams, but segmentation boundaries randomize across file boundaries affecting performance and deduplication effectiveness
Solution Approach 1:
The patent introduces boundary markers at the beginning and end of data regions as preliminary actions before the actual data processing. These markers预先 indicate file boundaries, allowing the storage system to perform effective deduplication and segmentation while maintaining continuous stream processing efficiency.
Solution Approach 2:
The boundary markers act as intermediaries between the continuous data stream and the file-based processing requirements. These markers provide structural information about file boundaries without interrupting the continuous data flow, enabling the storage system to maintain both processing efficiency and deduplication effectiveness.
3Reliability
If checksum data is stored in the data stream, then integrity verification is enabled, but the data stream size increases and deduplication space utilization decreases
Solution Approach 1:
The patent uses marker data structures that copy only the essential boundary information (start and end positions) rather than storing the entire checksum data repeatedly. This copying approach maintains integrity verification capability while minimizing the space occupied by metadata, thereby improving overall storage space utilization.
Data Source
AI summary
Techniques for deduplicating a data stream based on boundary markers embedded therein are described. According to one embodiment, a data stream is received from a client having a sequence of a plurality of data objects, where to data stream represents a file or a directory of one or more files of a file system associated with the client. In response, the data stream is deduplicated into a plurality of deduplicated chunks in view of boundaries of the data objects.