Hierarchical Storage System Using SHA-1 Signatures for Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage and retrieval systems face inefficiencies in reducing storage space and retrieval time due to the lack of effective methods for identifying and eliminating duplicate data sets, and they do not provide secure and encrypted data transfer without requiring the complete data set.
Innovation Solution
A hierarchical data storage and retrieval system utilizing statistically unique signatures, such as those computed by the SHA-1 algorithm, to identify and eliminate duplicate data sets, combined with encryption methods like XOR encryption to ensure secure data transfer by splitting data into multiple portions, requiring all portions for decryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data storage systems store complete data sets, then data retrieval is straightforward, but storage space is wasted due to duplicate data sets
Solution Approach 1:
The patent extracts only the essential identifying information (hash values) from complete data sets to create a catalog. This allows the system to store references to data rather than duplicates of entire data sets, dramatically reducing storage space while maintaining the ability to retrieve complete data when needed.
Solution Approach 2:
The patent creates a catalog that copies only the hash values and metadata of data sets rather than the actual data content. This catalog serves as an index that points to the original data storage locations, enabling efficient retrieval without storing redundant data copies.
2Loss of time
If traditional backup systems store complete data sets, then data can be restored, but retrieval time is increased due to processing large data volumes
Solution Approach 1:
The system extracts and stores only the hash values and metadata in the catalog, which are much smaller than the complete data sets. When retrieval is needed, the system uses these small catalog entries to quickly locate and retrieve only the required data portions, significantly reducing processing time and data volume handled.
3Reliability
If data is encrypted as a single unit, then decryption is simple, but security is compromised if any portion of the data is intercepted
Solution Approach 1:
The patent divides the data into separate portions, each encrypted with a unique key. This segmentation ensures that if one portion is intercepted or compromised, the others remain secure. The catalog stores references to these encrypted portions, allowing selective retrieval and decryption of only the needed data segments.
4Quantity of substance
If the system stores only unique data sets using hash functions, then storage space is reduced, but validation of data uniqueness becomes complex
Solution Approach 1:
The system performs preliminary hash computation on data before storage. By calculating and storing hash values in advance, the system can quickly compare hashes to determine uniqueness without having to process the entire data content during validation operations. This pre-computation simplifies the uniqueness detection process.
Data Source
AI summary
Disclosed herein is a hierarchical data storage and retrieval system implemented using a computed statistically unique signature for the content of given computer data as its basis. The data storage and retrieval system can be used to catalog computer information for easy indexing and retrieval. The signature is computed by any number of techniques so long as it is able to produce a statistically unique signature, one example being the SHA-1 algorithm. The system is able to reduce the amount of data that is stored and the time required for retrieving the data. Provide a way to move electronic data in an encrypted secure manner that requires the complete data set being moved for decryption of the data. Even if a portion of the data is compromised that portion would still remain secure. The data is encrypted and separated into two or more portions all of which would be needed to decrypt the data. Detailed information on various example embodiments of the inventions are provided in the Detailed Description below, and the inventions are defined by the appended claims.