Segment Deduplication with Pre-Encryption Hash Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing segment deduplication systems are inefficient when data is encrypted or compressed, as these processes prevent the identification of identical segments, thereby eliminating the efficiency of deduplication.
Innovation Solution
A system that compresses and encrypts data segments while maintaining the ability to identify and store only unique segments, using processors to determine if compressed or encrypted segments have been previously stored, and reconstructing data by decompressing and decrypting as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted prior to storage, then security is improved, but the ability to identify identical segments is lost, eliminating deduplication efficiency
Solution Approach 1:
The system performs deduplication identification on plaintext data before encryption occurs. By calculating hashes and identifying duplicate segments in the original unencrypted form, the system maintains deduplication efficiency while allowing subsequent encryption to provide security. This preliminary action resolves the contradiction by establishing duplicate detection before the encryption process obscures the data.
Solution Approach 2:
The system introduces an intermediary hashing mechanism that operates on plaintext data before encryption. The hash values serve as intermediaries that preserve the ability to identify identical segments without requiring the actual encrypted data to be compared. This intermediary approach allows both security (through encryption) and deduplication efficiency (through hash-based identification) to coexist.
2Loss of substance
If compression is applied to data, then bandwidth savings and storage efficiency are improved, but the ability to identify identical segments is lost, eliminating deduplication efficiency
Solution Approach 1:
The system performs deduplication identification on uncompressed plaintext data before compression is applied. By identifying and eliminating duplicate segments in the original form, the system maintains deduplication efficiency while allowing subsequent compression to provide bandwidth and storage savings. This preliminary action resolves the contradiction by establishing duplicate detection before the compression process alters the data structure.
Solution Approach 2:
The system introduces an intermediary hashing mechanism that operates on uncompressed plaintext data before compression. The hash values serve as intermediaries that preserve the ability to identify identical segments without requiring the actual compressed data to be compared. This intermediary approach allows both compression efficiency and deduplication efficiency to coexist.
3Reliability
If both encryption and compression are applied to data segments, then security and bandwidth savings are improved, but segment identification for deduplication is prevented
Solution Approach 1:
The system performs deduplication identification on plaintext data before both encryption and compression are applied. By calculating hashes and identifying duplicate segments in the original unencrypted and uncompressed form, the system maintains deduplication efficiency while allowing subsequent encryption and compression to provide security and bandwidth savings. This preliminary action resolves the contradiction by establishing duplicate detection before the processing steps obscure the data.
Solution Approach 2:
The system introduces an intermediary hashing mechanism that operates on plaintext data before encryption and compression. The hash values serve as intermediaries that preserve the ability to identify identical segments without requiring the actual encrypted and compressed data to be compared. This intermediary approach allows security, compression efficiency, and deduplication efficiency to coexist.
Data Source
AI summary
A system for storing compressed data comprises a processor and a memory. The processor is configured to receive a compressed segment. The compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing each of the one or more segments. The processor is further configured to determine whether the compressed segment has been previously stored, and in the event that the compressed segment has not been previously stored, store the compressed segment. The memory is coupled to the processor and configured to provide the processor with instructions.