Secure Data De-duplication via Cryptographic Hashing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Data de-duplication systems face concerns regarding data integrity and unauthorized access, particularly during the transformation, de-duplication, and access of data, due to potential hash collisions and increased vulnerability to malicious attacks.

Innovation Solution

A method and apparatus that utilize cryptographic protocols such as Diffie-Hellman and AES encryption, along with cyclic redundancy checksums, to securely process data for backup, de-duplication, and restoration, ensuring data integrity and security by using a de-duplication server, database query handler, and file registrar server to manage and encrypt data files, and maintain a registry of de-duplicated files.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data de-duplication is implemented to reduce storage capacity, then storage efficiency is improved, but data integrity and security are compromised due to potential hash collisions and unauthorized access

Engineering Contradiction:

Improvestorage capacityVSAvoiddata integrity

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent segments data into fixed-size chunks and computes cryptographic hash values for each chunk. This segmentation allows de-duplication at the chunk level while maintaining data integrity through cryptographic verification. Each data chunk is independently hashed and stored with its hash value, enabling secure identification without storing redundant copies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent replaces traditional mechanical de-duplication methods with cryptographic hash functions. Instead of relying on file system-level duplication detection, the system uses cryptographic hashing (SHA-256 or similar) to identify and eliminate redundant data chunks. This substitution provides stronger integrity guarantees through the mathematical properties of cryptographic hash functions.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Quantity of substance

If data is transformed and de-duplicated to increase storage utilization, then storage efficiency is improved, but vulnerability to malicious attacks increases

Engineering Contradiction:

Improvestorage utilizationVSAvoidmalicious attacks

Core Design Contradiction:

Quantity of substanceVSObject-affected harmful factors

Solution Approach 1:

The patent applies preliminary anti-action by encrypting data chunks with AES-256 encryption before storing them in the de-duplicated repository. This pre-encryption protects the data from malicious attacks during storage and transmission. The cryptographic protection is established before any de-duplication operations occur, ensuring that even if data is transformed, it remains secured against unauthorized access and tampering.

Inventive Principle:

Principle #9Preliminary anti-action

Solution Approach 2:

The patent introduces cryptographic hash values as intermediaries between the original data and the de-duplicated storage. Instead of directly storing and comparing raw data chunks, the system uses hash values as mediators for identification and verification. This intermediary layer enables secure de-duplication while maintaining data integrity and protecting against malicious attacks through the mathematical properties of cryptographic hashing.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If cryptographic hash functions are used to identify duplicate segments, then de-duplication accuracy is improved, but the risk of data corruption increases due to hash collisions

Engineering Contradiction:

Improveduplicate identification accuracyVSAvoiddata corruption risk

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent changes the parameter of hash function selection to use cryptographic-grade hash functions (such as SHA-256) with sufficiently large output sizes. This parameter change reduces the probability of hash collisions to negligibly small values, effectively eliminating the data corruption risk while maintaining high de-duplication accuracy. The system may also use multiple hash functions or combine hash values with additional verification mechanisms.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent implements beforehand cushioning by storing both the cryptographic hash value and the original data chunk (or a verified copy) in the de-duplicated repository. This redundancy acts as a cushion against potential hash collisions, allowing the system to verify data integrity by comparing the stored hash with a recomputed hash of the stored data. If a collision is detected, the system can identify and correct the error before it causes data corruption.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS9122882B2Method and apparatus of securely processing data for file backup, de-duplication, and restoration

Publication Date: 2015.09.01 KASEYA

AI summary

Disclosed are an apparatus and method of de-duplicating at least one data file. One example method may include requesting a list of data rows stored in a database to be de-duplicated, receiving the list of the data rows based on a single filekey associated with the at least one data file, copying the at least one data file to a data storage memory, and deleting the data rows entries from a file registry of the database.