Scalable Deduplication via Centralized Hash Enforcement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication mechanisms are not scalable to multiple devices concurrently identifying and eliminating data redundancies in real-time across a network, leading to potential duplication of search results during concurrent investigations.
Innovation Solution
A method involving a central database with a deduplication table that uses content hash and metadata hash fields, along with a level field, to enforce uniqueness across multiple devices, ensuring that only unique data items are added, preventing duplication by comparing hash values and metadata across devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple devices concurrently conduct investigations and store data items independently, then each device can operate autonomously and maintain local data integrity, but duplicate data items will be stored across multiple devices leading to redundant processing and wasted storage space
Solution Approach 1:
The patent merges deduplication functionality across multiple devices by implementing a centralized deduplication server that coordinates hash value storage and comparison. Multiple examining machines share a common deduplication database, allowing them to collectively identify and eliminate duplicate data items while maintaining autonomous investigation capabilities.
Solution Approach 2:
The deduplication server acts as an intermediary between multiple examining machines and the central database. It receives hash values from examining machines, performs deduplication operations, and returns results, thereby preventing redundant data storage without interfering with the autonomous operation of individual devices.
2Loss of substance
If a centralized deduplication mechanism is implemented across multiple devices, then duplicate data items can be eliminated and storage efficiency improved, but system complexity and network communication overhead increase
Solution Approach 1:
The patent replaces complex mechanical coordination between multiple devices with a streamlined network-based information exchange system. Instead of devices directly communicating and coordinating with each other, they all interact with a centralized deduplication server through standardized hash value transmissions, simplifying the overall system architecture.
Solution Approach 2:
The system changes the parameter of data representation by using hash values instead of actual data items for comparison and storage. This transformation allows the deduplication mechanism to operate efficiently with minimal data transmission and storage requirements, reducing system complexity while maintaining effectiveness.
3Loss of substance
If deduplication checks are performed before data storage, then duplicate data can be identified and eliminated, but processing time increases due to additional lookup operations
Solution Approach 1:
The patent performs deduplication checks as a preliminary action before data items are stored in the central database. By computing hash values and checking for duplicates prior to storage, the system prevents redundant data from being written to disk, thereby eliminating future retrieval and processing overhead despite the initial time cost.
Solution Approach 2:
The system uses hash values as copies or representations of actual data items for deduplication purposes. Instead of comparing entire data items, which would be time-consuming, the system compares compact hash value representations, dramatically reducing processing time while maintaining accurate duplicate detection.
Data Source
AI summary
A system and method for data deduplication includes a first computer device that determines duplicacy of a data item. If the data item is not a duplicate, the first computer device transmits a request to add an entry for the data item in a deduplication table of a deduplication database. The database adds the entry for the data item while enforcing uniqueness of data across one or more data fields of the deduplication table, where, in enforcing the uniqueness, the database denies an attempt by the second device to add an entry in the deduplication table for the same data item.