Single-Ended Deduplication Protocol for Cloud Storage Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer methods to cloud storage services over wide-area networks are inefficient due to bandwidth limitations, requiring expensive double-ended WAN optimization devices or storing data in compressed form, which burdens clients with decompression needs.
Innovation Solution
A single-ended optimized storage protocol that allows a storage client to direct a remote data storage to copy and reconstruct data streams without transferring data back for processing, removing redundant segments and storing data in its native form, thus reducing bandwidth and time requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If double-ended WAN optimization devices are used to compress data at source and decompress at destination, then data transfer bandwidth and time are improved, but device complexity and cost increase significantly
Solution Approach 1:
The invention extracts the decompression function from the cloud storage service side and relocates it to the storage client side. Only the compression function remains at the source, eliminating the need for expensive double-ended optimization devices while maintaining improved data transfer rates.
Solution Approach 2:
The invention introduces an intermediary mechanism where the storage client performs decompression locally and sends only the decompressed necessary portions to the cloud storage service. This mediator approach reduces the complexity at the destination by eliminating the need for decompression capabilities at the cloud storage service.
2Loss of energy
If data is stored in compressed form in the cloud storage service, then bandwidth usage is reduced, but data usability and ease of operation worsen due to decompression requirements
Solution Approach 1:
The invention performs preliminary decompression at the storage client side before uploading to the cloud storage service. By decompressing data beforehand and sending only the necessary portions in uncompressed form, the system reduces bandwidth consumption during subsequent operations while ensuring data is immediately usable without additional decompression requirements.
3Quantity of substance
If all data is transferred to cloud storage service over WAN, then storage capacity is utilized, but transfer time increases due to bandwidth limitations
Solution Approach 1:
The invention extracts redundant data segments from the data stream before transfer to the cloud storage service. By identifying and removing duplicate or unnecessary portions locally, the system reduces the total volume of data that needs to be transferred over the WAN, thereby utilizing storage capacity effectively while minimizing transfer time.
4Productivity
If WAN optimization devices are deployed at both client and cloud storage service locations, then data transfer efficiency is improved, but cost and device complexity increase
Solution Approach 1:
The invention extracts the decompression functionality from the cloud storage service infrastructure and implements it at the storage client side. This eliminates the need for expensive WAN optimization devices at the cloud storage service location, maintaining data transfer efficiency while significantly reducing overall system deployment cost.
Solution Approach 2:
The storage client performs decompression operations itself rather than relying on external optimization devices at the cloud storage service. This self-service approach eliminates the need for costly infrastructure at the service provider end while maintaining high data transfer efficiency through local processing.
Data Source
AI summary
A single-ended optimized storage protocol enables storage clients or other devices to direct a remote data storage to copy data. In response to commands via the protocol, a remote data storage can copy portions of a data stream at the remote data storage to destination storage locations within the same or a different data stream. The protocol may be utilized for optimized transfer of data via a network to a remote data storage. An initial data stream is divided into segments. Redundant segments are removed from the data stream to form an optimized data stream, which is transferred to the remote data storage. Commands are issued to the remote data storage using the protocol to direct the remote data storage to reconstruct the initial data stream at the remote data storage using the optimized data stream and optionally segments from other data streams previously transferred to the remote data storage.