Inline Deduplication Storage Appliance Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual tape libraries face inefficiencies in data deduplication due to offline deduplication processes that require additional disk storage and I/O cycles, leading to higher costs and performance penalties, especially when handling large volumes that exceed main memory capacity.
Innovation Solution
The storage appliance segments data into manageable sizes that fit within main memory, allowing for inline deduplication processing without disk I/O, using a deduplication engine to reduce resource utilization and enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If offline deduplication is used to handle large volumes exceeding main memory capacity, then data deduplication can be performed, but additional disk storage resources and I/O cycles are required, leading to higher costs and performance penalties
Solution Approach 1:
The patent segments large volumes exceeding main memory capacity into smaller chunks that can be processed in parallel. The deduplication engine divides incoming data streams into manageable segments, processes each segment independently in main memory, and combines the results. This segmentation enables inline deduplication of large volumes without requiring disk I/O, thus maintaining high storage performance while achieving comprehensive data deduplication.
2Quantity of substance
If offline deduplication is performed by reading data from disk, applying deduplication algorithm and storing reduced data, then data deduplication is achieved, but additional disk storage resources and I/O cycles are required
Solution Approach 1:
The patent implements continuous inline deduplication processing where data is deduplicated as it flows through the system rather than being read from disk in batches. The deduplication engine maintains continuous operation on incoming data streams, processing segments in real-time through main memory and immediately storing only unique data blocks. This eliminates the stop-start nature of offline deduplication and removes the need for additional disk I/O cycles, achieving both high deduplication ratios and energy efficiency.
3Quantity of substance
If virtual volumes with capacities up to 200 GB are handled, then large data sets can be stored, but they will not fit into main memory when a large number of volumes are handled in parallel
Solution Approach 1:
The patent segments large virtual volumes into smaller processing chunks that fit within main memory capacity. Each volume is divided into multiple segments that can be independently processed in parallel. The system manages multiple segments across available memory resources, loading only the necessary portions into main memory at any given time. This segmentation approach enables handling of large 200 GB volumes while maintaining manageable memory complexity through systematic division and parallel processing.
Solution Approach 2:
The patent implements dynamic memory allocation and segment loading strategies where the system adaptively manages main memory resources based on current workload demands. When handling multiple large volumes in parallel, the system dynamically loads and unloads segments as needed, optimizing memory utilization. This dynamic approach allows flexible handling of variable volume sizes and quantities without requiring static over-provisioning of memory resources, thus managing complexity effectively.
Data Source
Figure 1
Figure 2A~2C
Figure 3
AI summary
Ingest data for virtual volumes (V) is split into segments (B1, B2, B3, B4) of a size that can be buffered in main memory. Data deduplication processing then occurs directly on the segments (B1, B2, B3, B4) in main memory, without the need for disk I/O.