Data Deduplication via Multi-Level Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data platforms, especially big data platforms, there is a need to address data redundancy and inefficiencies caused by duplicate data across multiple data stores, leading to increased storage costs and operational inefficiencies, without visibility into data redundancy and usage patterns.
Innovation Solution
A method and system for data deduplication storage management that determines multi-level signature mappings of data content across data stores, assesses data similarity, and tracks usage to detect and eliminate duplicate data, optimizing storage by compressing data based on usage patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored across multiple data stores in a data platform, then data accessibility and availability are improved, but data redundancy and storage costs increase
Solution Approach 1:
The patent uses content-based copying where duplicate data copies are created across multiple data stores based on content similarity analysis. The system generates content signatures for data objects and identifies duplicates by comparing signatures, allowing selective copying only when content matches, thereby improving accessibility while controlling redundancy through intelligent duplicate detection
Solution Approach 2:
The patent changes the parameter of data identification from location-based to content-based by generating content signatures (hashes) of data objects. This parameter change enables the system to identify duplicate data regardless of its storage location, allowing intelligent deduplication decisions that balance accessibility needs with redundancy reduction
2Quantity of substance
If data deduplication is implemented across multiple data stores, then storage efficiency is improved, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the data deduplication process into distinct phases: content signature generation, signature comparison, duplicate identification, and deduplication execution. This segmentation allows the system to manage computational complexity by processing data in manageable stages rather than attempting simultaneous analysis of all data across all stores
Solution Approach 2:
The patent introduces content signatures as an intermediary mechanism that simplifies duplicate detection. Instead of directly comparing large data objects, the system compares compact signature representations (hashes) of the data. This intermediary approach dramatically reduces computational complexity while maintaining accurate duplicate identification
3Measurement precision
If multi-level signature mapping is used to detect duplicate data, then duplicate detection accuracy is improved, but processing overhead and time increase
Solution Approach 1:
The patent segments data into hierarchical levels (data objects, data blocks, data segments) and generates content signatures at multiple levels. This multi-level segmentation allows the system to first compare high-level signatures to quickly eliminate non-duplicates, then progressively dive deeper into lower levels only for potential matches, thereby maintaining high accuracy while reducing overall processing time through early elimination
Data Source
AI summary
One embodiment of the invention provides a method for data deduplication storage management in a data platform including a plurality of data stores. The method comprises, for each data store of the plurality of data stores, determining a corresponding multi-level signature mapping data content of the data store into an ordered logical form comprising a plurality of data abstraction levels, determining a data similarity between the data store and each other data store of the plurality of data stores based on the multi-level signature corresponding to the data store and another multi-level signature corresponding to the other data store, and determining data usage of the data content of the data store. The method further comprises improving storage in the data platform by detecting duplicate data across the plurality of data stores based on each data similarity determined and each data usage determined.