Scalable Grid Deduplication for Backup Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deduplication-based data storage systems face availability issues due to reliance on a single compute server for deduplication processing, leading to increased end-to-end backup times as data volume grows, exceeding service level agreement limits.
Innovation Solution
A scalable server grid system is implemented, where each grid server includes disk storage, processing units, memory, and networking ports, enabling distributed deduplication processing across multiple servers, with failover/recovery models to maintain availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single compute server is used for deduplication processing, then device complexity is reduced, but reliability deteriorates due to single-point failure
Solution Approach 1:
The patent divides the deduplication processing system into multiple independent compute servers organized in a grid architecture. Each server handles a portion of the deduplication workload, eliminating the single-point failure problem while distributing system complexity across manageable units. The grid structure allows individual servers to be added, removed, or failed without affecting the entire system.
Solution Approach 2:
The patent combines multiple compute servers with storage units into an integrated grid system where processing and storage resources work together. This merging creates a more reliable system through redundancy while managing complexity through unified resource orchestration and coordination among grid members.
2Quantity of substance
If additional disk storage units are added to handle increased data volume, then storage capacity increases, but productivity deteriorates as end-to-end backup time exceeds service level agreement limits
Solution Approach 1:
The patent segments the deduplication processing workload across multiple compute servers in the grid, allowing parallel processing of deduplication tasks. This segmentation enables the system to handle increased storage capacity requirements while maintaining backup speed within service level agreement limits through distributed computation.
Solution Approach 2:
The patent implements a dynamic grid system where compute servers can be added or removed based on workload demands and capacity requirements. This dynamic architecture allows the system to scale processing power proportionally with storage capacity, ensuring backup performance adapts to changing data volumes while meeting service level agreements.
3Device complexity
If a single compute server performs all deduplication processing, then device complexity is minimized, but productivity deteriorates as processing capacity becomes a bottleneck
Solution Approach 1:
The patent divides deduplication processing into multiple independent compute servers within a grid architecture. Each server handles a portion of the processing workload, collectively providing high processing capacity while maintaining manageable complexity through modular design and distributed operation.
Solution Approach 2:
The patent creates a universal grid system where compute servers can perform multiple functions including deduplication processing, data management, and coordination tasks. This multi-functionality increases overall productivity while managing complexity through standardized server roles and interchangeable components within the grid.
Data Source
AI summary
A system, a method, and a computer program product for performing deduplication of data using a scalable deduplication grid are disclosed. A listing of a plurality of zone stamps is generated, where each zone stamp represents a zone in the plurality of zones in a data stream. The listing contains a logical arrangement of the plurality of zone stamps obtained from each storage location and being accessible by a plurality of servers. A first zone stamp in the listing is compared to a second zone stamp in the listing. The first and second zones are delta-compressed based on a determination that the first zone stamp is substantially similar to the second zone stamp. A server is selected to perform the comparison and delta-compression.