Server-Side Data De-Duplication Using Slepian-Wolf Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication systems face challenges in reducing bandwidth requirements, particularly due to the inefficiencies of client-side data de-duplication and the need for large local caches, and standard compression methods perform poorly with hash-based data de-duplication, leading to lower compression ratios.
Innovation Solution
The method involves encoding data blocks into bitstreams using Slepian-Wolf encoding, allowing for efficient decoding and de-duplication on a server device, with the ability to adjust transmission rates and handle incomplete decodings by requesting additional information from the client, thereby reducing redundant data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If client side data de-duplication is implemented, then bandwidth requirements are reduced, but deployment difficulty increases and direct compare becomes difficult
Solution Approach 1:
The patent introduces an intermediary compression process that transforms original data into a compressed representation before transmission. This intermediary step enables server-side de-duplication to work effectively without requiring client-side implementation, thus reducing bandwidth usage while maintaining deployment simplicity.
2Productivity
If standard compression methods are applied to hash-based data, then de-duplication can be performed, but compression ratios deteriorate
Solution Approach 1:
The patent applies parameter changes by transforming the data representation from original format to a compressed format that preserves de-duplication capabilities. The compression process modifies parameters such as data encoding and structure, enabling both high compression ratios and effective de-duplication on hash-based data.
3Productivity
If smaller chunk sizes are used for de-duplication, then compression ratios improve, but index metadata size increases
Solution Approach 1:
The patent performs preliminary compression of data before the de-duplication process. This preliminary action reduces the overall data size and consequently reduces the size of index metadata required, while still allowing smaller chunk sizes to be used for achieving good compression ratios during de-duplication.
Data Source
AI summary
The invention provides a method and system for reducing redundant data blocks. The method includes reducing redundant data blocks by decoding a first data block from a first plurality of data blocks and a bitstream.