Data Segment Sketching for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data backup volumes increase, existing methods for efficient storage face a trade-off between access performance and storage efficiency, where maintaining access performance leads to reduced storage efficiency due to increased segments and index entries, and increasing segment size decreases index entries, compromising performance.
Innovation Solution
The approach involves segmenting data streams into logical segments, determining resemblance using summary feature sets, and representing segments as composites of base and delta segments for efficient storage and retrieval, utilizing one-level or two-level delta coding to balance compression and performance, while maintaining a mapping between address ranges and segment descriptors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the number of segments is increased to maintain access performance, then storage efficiency drops because there are not as many occurrences of the longer identical segments
Solution Approach 1:
The patent divides data streams into segments of optimized size, creating a balance between the number of segments and their individual sizes. This segmentation allows the system to maintain access performance while maximizing the likelihood of finding identical segments for deduplication, thereby resolving the contradiction between access speed and storage efficiency.
Solution Approach 2:
The patent dynamically adjusts segment size as a parameter to optimize both access performance and storage efficiency. By changing the segment size parameter based on data characteristics and access patterns, the system can maintain good access performance while ensuring sufficient storage efficiency through appropriate segment granularity.
2Device complexity
If the size of the segments is increased, then the number of segments decreases and therefore the number of index entries also decreases and performance can be maintained; however, the storage efficiency drops because there are not as many occurrences of the longer identical segments
Solution Approach 1:
The patent optimizes segment size as a key parameter to achieve the right balance. By carefully selecting segment sizes that are not too large (which would reduce storage efficiency) and not too small (which would increase index complexity), the system maintains both good access performance and storage efficiency simultaneously.
Solution Approach 2:
The patent uses indexing to create references to segment copies rather than storing all segments fully. This copying approach reduces the number of actual index entries needed while maintaining efficient access performance, as the index points to reusable segment copies in storage.
Data Source
AI summary
Determining a summary feature set is disclosed. A plurality of subsegments of a first segment are selected. For each subsegment, a plurality of values by applying a set of functions to each subsegment are computed. From all the values computed for all the subsegments, a first subset of values is selected.