Server-Side Data De-Duplication Using Slepian-Wolf Bitstreams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication systems face challenges in reducing bandwidth requirements, particularly due to the inefficiencies of client-side de-duplication deployment and the poor performance of standard compression methods on hash-based data, leading to lower compression ratios and increased data transmission.
Innovation Solution
The method involves encoding data blocks into bitstreams using Slepian-Wolf encoding, allowing for efficient decoding and de-duplication on a server device, with the option to request additional information from the client if decoding is unsuccessful, and adjusting the transmission rate to meet a predetermined collision rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is transmitted before de-duplication at the server, then server side de-duplication can be performed, but bandwidth requirements from client to server increase
Solution Approach 1:
The patent applies preliminary action by performing encoding of data blocks into bitstreams at the client side before transmission, preparing the data in advance to enable efficient server-side de-duplication. This preliminary encoding reduces the amount of data that needs to be transmitted while maintaining the ability to perform de-duplication at the server.
2Quantity of substance
If client side data de-duplication is performed, then bandwidth requirements are reduced, but deployment difficulty increases due to tighter integration requirements
Solution Approach 1:
The patent applies segmentation by dividing the de-duplication process into two distinct parts: encoding at the client side and decoding with de-duplication at the server side. This segmentation allows the system to achieve bandwidth reduction benefits while maintaining server-side deployment simplicity, as each side performs its designated function independently without requiring tight integration.
3Productivity
If smaller chunk sizes are used for de-duplication, then compression ratios improve, but index metadata size increases
Solution Approach 1:
The patent applies the taking out principle by extracting only the essential information needed for de-duplication to the server side (the bitstream representation), while the client side handles the encoding process. This extraction allows smaller chunk sizes to be used for better compression ratios without proportionally increasing the metadata burden, as the encoding process efficiently represents the data.
4Productivity
If standard compression methods are applied to hash lists, then compression is attempted, but performance is poor
Solution Approach 1:
The patent applies parameter changes by transforming the data representation from traditional hash lists to bitstream encodings of data blocks. This fundamental parameter change in how data is represented and processed enables significantly better compression performance compared to standard compression methods applied to hash lists, while the encoding process itself handles the complexity efficiently.
Data Source
AI summary
The invention provides a method and system for reducing redundant data blocks. The method includes encoding a first data block having a first length into a bitstream having a second length, transmitting the bitstream to a server device, and reducing redundant data blocks by decoding the first data block from a first plurality of data blocks and the bitstream where each block in the first plurality of data blocks has a length equal to the first length.