Deduplication Appliance Inversion for Network Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication methods require clients to send data to a deduplication appliance for redundancy removal, leading to high network bandwidth and CPU usage, especially during data copy operations, which is cumbersome and inefficient.
Innovation Solution
A deduplication appliance proactively accesses and deduplicates data stored locally on client computers, replacing actual data with pointers or stub files, allowing clients to access deduplicated data without moving it across the network, and implementing a 'dcopy' command to manage copying requests by creating stub files for deduplicated data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If clients send data to the deduplication appliance for deduplication, then data redundancy is removed, but network bandwidth usage and CPU usage increase significantly
Solution Approach 1:
Instead of clients sending data to the deduplication appliance for deduplication, the appliance proactively mounts and accesses data stored locally on client computers, performing deduplication at the source. This inversion of the data flow direction eliminates the need to move actual data across the network, thereby reducing network bandwidth usage and CPU usage while still achieving redundancy removal.
Solution Approach 2:
The deduplication appliance proactively mounts network shared storage devices on client computers and accesses data before clients need to use it. By performing deduplication operations in advance and replacing data with stub files containing indicators, the system prepares the data structure for efficient access without requiring data movement during actual client operations.
2Loss of substance
If data is moved across the network for deduplication, then redundancy is identified, but network bandwidth is consumed
Solution Approach 1:
The system inverts the traditional data movement approach by having the deduplication appliance access and process data locally stored on client computers rather than moving data to the appliance. This eliminates network bandwidth consumption for data transfer while still enabling redundancy identification through local hash computation and comparison.
Solution Approach 2:
The deduplication appliance creates stub files containing indicators (pointers) that reference the actual data locations. These stub files are copied to client systems instead of moving the actual data, allowing the appliance to maintain references to redundant data without consuming network bandwidth for data transfer.
3Ease of operation
If the entire data set is retrieved for copy operations, then data management is performed, but CPU usage and network bandwidth increase
Solution Approach 1:
The system implements a dcopy command that operates on stub files and indicators rather than actual data. By performing copy operations at the metadata level (copying indicators and stub files) rather than retrieving and transferring entire data sets, the system enables data management operations with minimal CPU usage and network bandwidth consumption.
4Ease of operation
If data is stored locally on client computers, then accessibility is maintained, but storage efficiency is reduced
Solution Approach 1:
The system segments data representation into two parts: stub files stored locally on client computers that provide accessibility, and actual data stored on the deduplication appliance. The stub files contain indicators that reference the actual data locations, allowing clients to access data efficiently while the appliance maintains deduplicated storage of the actual data blocks.
Solution Approach 2:
Instead of storing full data copies locally, the system creates and stores stub files containing indicators (pointers) that reference the actual data. These stub files are lightweight copies that maintain client accessibility while enabling the deduplication appliance to store only unique data blocks centrally, thereby improving storage efficiency without sacrificing access convenience.
Data Source
AI summary
A method of deduplicating data is disclosed comprising mounting, by a deduplication appliance, network shared storage of a client machine, via a network, accessing data to be deduplicated on the network shared storage device, deduplicating the data, storing the deduplicated data on a second storage device, and replacing the data in the network shared storage device by at least one indicator of the location of the deduplicated data in the second storage device. A method is also disclosed for copying deduplicated data stored by a deduplication appliance, by a client machine, comprising receiving a request to copy data from a first location to a second location, by the client machine, by providing at least one second indicator to the third location on the deduplication appliance, at the second location, if the source and the destination are on the deduplication appliance. Systems are also disclosed.