Data Segment Resemblance Storage for Index Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data volume increases, existing methods for efficient data storage through segmentation and indexing lead to reduced access performance due to increased number of segments and index entries, compromising both access speed and storage efficiency.
Innovation Solution
The method involves breaking data streams into logical segments, representing them as composites of shared base and delta segments, and comparing summary feature sets to determine resemblance, storing new segments based on deltas, which enhances storage efficiency by reducing redundant data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If the number of segments is increased to maintain storage efficiency, then storage efficiency is improved, but access performance deteriorates due to increased number of index entries
Solution Approach 1:
The patent divides the index into multiple sub-indices, each containing a subset of segment identifiers. This segmentation of the index structure allows parallel processing during search operations, reducing the time to scan through all segment identifiers while maintaining fine-grained segment tracking for storage efficiency
Solution Approach 2:
The patent introduces a hierarchical indexing structure with multiple levels, where the first level contains sub-index identifiers and the second level contains actual segment identifiers. This dimensional transformation converts a single large linear search into a multi-level hierarchical search, improving access performance while maintaining storage efficiency
2Speed
If the size of segments is increased to maintain access performance, then access performance is improved, but storage efficiency deteriorates because there are not as many occurrences of longer identical segments
Solution Approach 1:
The patent divides data into fine-grained segments that can be independently identified and stored. By creating smaller segment units, the system increases the probability of finding identical segments across different backups, improving storage efficiency through deduplication while maintaining fast access through efficient indexing
Solution Approach 2:
The patent replaces traditional sequential scanning of segment identifiers with a hash-based indexing mechanism. The hash function quickly maps segment data to specific index locations, eliminating the need to linearly search through all segment identifiers, thus maintaining fast access performance with fine-grained segmentation
3Loss of substance
If the number of index entries is increased to track more segments, then storage efficiency is improved, but device complexity increases and access performance deteriorates
Solution Approach 1:
The patent segments the index into multiple sub-indices, distributing segment identifiers across different sub-indices. This segmentation reduces the complexity of each individual sub-index while maintaining comprehensive tracking of all segments, improving both manageability and access performance
Solution Approach 2:
The patent introduces sub-index identifiers as intermediary elements between the main index structure and individual segment identifiers. This intermediary layer simplifies the overall index structure by organizing segments into manageable groups, reducing complexity while maintaining storage efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Storage using resemblance of data segments is disclosed. It is determined that a new segment resembles a prior stored segment. The prior stored segment comprises a segment stored previously from any location in an input data stream. A delta between the new segment and the prior stored segment is determined. A representation of the new segment based at least in part on the delta is stored.