Synchronized Data Deduplication via Reference Table Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data in networked computing environments leads to bandwidth overload and storage challenges, as redundant data is not efficiently managed, causing congestion and strain on backup and recovery systems, despite advancements in storage capacity.
Innovation Solution
Implementing a system for data deduplication that shares a reference table among client systems, allowing local deduplication before data is sent to a shared storage repository, and transferring only subsets of the reference table based on data utilization criteria to reduce bandwidth consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored and shared by networked devices, then storage capacity is increased, but network bandwidth becomes overloaded
Solution Approach 1:
The system performs preliminary deduplication actions by creating and distributing reference tables to client systems before data is fully transferred. Clients use these reference tables to identify and eliminate redundant data blocks locally, performing the deduplication work in advance before network transmission, thus reducing the actual data volume that consumes bandwidth.
Solution Approach 2:
Reference tables serve as intermediary structures that mediate between the storage repository and client systems. These reference tables contain information about data blocks already stored in the repository, allowing clients to determine whether data needs to be transferred without actually transmitting the data itself, thus reducing network bandwidth consumption.
2Reliability
If redundant data is stored multiple times, then data availability is improved, but storage efficiency deteriorates
Solution Approach 1:
The system implements feedback mechanisms where client systems report data block information to the storage repository, which maintains reference tables. These reference tables are then distributed back to clients, creating a feedback loop that enables continuous identification and elimination of redundant data blocks across the network, improving storage efficiency while maintaining data availability through the reference information.
Solution Approach 2:
Instead of copying actual data blocks multiple times across the network, the system creates and distributes copies of reference tables containing metadata about stored data blocks. This allows clients to have local copies of the reference information needed to identify redundancies without duplicating the actual data content, thus improving storage efficiency while maintaining data availability.
3Measurement precision
If reference tables are synchronized among all client systems, then deduplication effectiveness is improved, but bandwidth consumption increases
Solution Approach 1:
The system segments the reference table distribution process by sending different portions or updates of reference tables to different client systems based on their specific needs and the changes made to the repository. Instead of synchronizing entire reference tables with all clients, the reference table is divided and distributed selectively, maintaining deduplication effectiveness while reducing overall bandwidth consumption across the network.
Solution Approach 2:
The system applies local quality optimization by tailoring reference table distributions to individual client systems based on their local data characteristics and needs. Each client receives reference table information most relevant to its local data set, rather than a uniform distribution to all clients. This localized approach maintains effective deduplication at each client while minimizing total bandwidth consumption across the network.
Data Source
AI summary
A system and method for data deduplication is presented. Data received from one or more computing systems is deduplicated, and the results of the deduplication process stored in a reference table. A representative subset of the reference table is shared among a plurality of systems that utilize the data deduplication repository. This representative subset of the reference table can be used by the computing systems to deduplicate data locally before it is sent to the repository for storage. Likewise, it can be used to allow deduplicated data to be returned from the repository to the computing systems. In some cases, the representative subset can be a proper subset wherein a portion of the referenced table is identified shared among the computing systems to reduce bandwidth requirements for reference-table synchronization.