Segment Deduplication with Pre-Hashing for Compressed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing segment deduplication systems are inefficient when data is encrypted or compressed, as these processes prevent the identification of identical segments, thereby reducing the effectiveness of deduplication.
Innovation Solution
A system that compresses and/or encrypts data segments before storage, using processors to determine if the segments have been previously stored, and if not, stores them while maintaining the ability to identify and reconstruct the original data stream or file by using compatible encryption and compression systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is encrypted before storage, then security is improved, but the ability to identify identical segments is lost, reducing deduplication efficiency
Solution Approach 1:
The patent segments data into fixed-size blocks before encryption, allowing identical segments to be identified through hashing even after encryption. This segmentation enables the system to maintain deduplication efficiency while applying encryption for security.
Solution Approach 2:
The patent performs hashing on data segments before encryption to create unique identifiers. This preliminary action allows the system to identify identical segments through hash comparison even after encryption transforms the actual data, resolving the contradiction between security and deduplication efficiency.
2Quantity of substance
If compression is applied to data before storage, then storage efficiency is improved, but the ability to identify identical segments is lost, reducing deduplication effectiveness
Solution Approach 1:
The patent performs hashing on data segments before compression to create unique identifiers. This preliminary action enables the system to identify identical segments through hash comparison even after compression alters the data representation, maintaining deduplication effectiveness while achieving storage efficiency.
Solution Approach 2:
The patent introduces hashing as an intermediary mechanism between data segmentation and compression/encryption. This intermediary creates stable unique identifiers that remain valid even when the actual data undergoes compression or encryption, resolving the contradiction between storage efficiency and deduplication effectiveness.
3Reliability
If both encryption and compression are applied to data segments, then security and storage efficiency are improved, but segment identification becomes impossible, eliminating deduplication benefits
Solution Approach 1:
The patent performs hashing on data segments before both compression and encryption operations. This preliminary action creates unique identifiers that enable segment identification even after the data undergoes both compression and encryption, maintaining deduplication benefits while achieving both security and storage efficiency.
Solution Approach 2:
The patent introduces hashing as an intermediary mechanism that operates before compression and encryption. This intermediary creates stable unique identifiers that remain valid through both compression and encryption transformations, resolving the contradiction between security/storage efficiency and processing complexity.
Data Source
AI summary
A system for storing compressed data comprises a processor and a memory. The processor is configured to receive a compressed segment. The compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing each of the one or more segments. The processor is further configured to determine whether the compressed segment has been previously stored, and in the event that the compressed segment has not been previously stored, store the compressed segment. The memory is coupled to the processor and configured to provide the processor with instructions.