Hash Key De-duplication System for Backup Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for retrieving and managing data from large quantities of backup tapes are costly, inefficient, and prone to duplication, especially as systems evolve and data sensitivity increases, leading to challenges in recreating backup environments and storing vast amounts of redundant data.
Innovation Solution
A high-speed de-duplication system comprising pods and a database system that traverses data files, generates hash keys, and creates a Master Key table to eliminate redundant data by comparing file content and metadata, using SHA1 hashing and digital signatures to ensure non-redundant data storage across disparate locales.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional backup environment recreation methods are used to retrieve data from backup tapes, then data can be restored, but the process becomes prohibitively costly and complex as systems age and change
Solution Approach 1:
The system segments the backup environment into discrete components: hash key generation, Master Key table creation, and data file traversal. Each component operates independently, allowing the system to process backup data without recreating the entire original environment, thus reducing complexity while maintaining restoration capability
Solution Approach 2:
The patent introduces hash keys and a Master Key table as intermediary structures between the backup tapes and the restoration process. These intermediaries enable data identification and retrieval without requiring direct recreation of the original backup environment, simplifying the overall system
2Reliability
If all records from backup tapes are examined and stored to ensure complete data retrieval, then no data is lost, but storage space and computer resources are excessively consumed
Solution Approach 1:
The system changes the parameter of data representation by using hash keys (compact numerical identifiers) instead of storing complete data files. This parameter transformation allows the system to track and retrieve all unique records while consuming minimal storage space, as hash keys are vastly smaller than the actual data they represent
3Reliability
If backup frequency is increased to handle sensitive data, then data protection improves, but the amount of duplicative data stored increases significantly
Solution Approach 1:
The system performs preliminary hashing and Master Key table creation before actual data restoration. This preliminary action enables the system to identify and eliminate duplicates in advance, allowing frequent backups to be performed without proportionally increasing storage consumption, as the deduplication mechanism is already in place
Solution Approach 2:
The patent implements a mechanism to discard duplicative data by comparing hash keys against the Master Key table, keeping only unique records. This allows the system to perform frequent backups for data protection while automatically discarding redundant copies, thus reducing overall storage requirements
4Quantity of substance
If a single central system traverses all duplicated data to create a unique list, then data de-duplication is achieved, but the process becomes slow and taxing on system resources
Solution Approach 1:
The patent segments the de-duplication process into distinct phases: hash key generation, Master Key table population, and data file traversal. This segmentation allows each phase to be optimized independently and enables parallel processing possibilities, improving overall productivity compared to a single sequential traversal approach
Data Source
AI summary
The present invention is directed to a system and method for creating a non-redundant data set from a plurality of data sources. Generally, the system and method operate by creating unique hash keys corresponding to unique data files; compiling the hash keys along with seeking information for the corresponding data files; de-duplicating the hash keys; and retrieving/storing the data files corresponding to the de-duplicated hash keys. Thus, in accordance with the system and method of the present invention, a non-redundant data set can be created from a plurality of data sources. The system of the present invention can operate independently or in conjunction with any de-duplicating methods and systems. For example, a de-duplicating method and system can be used to read and obtain data from a variety of media, regardless of the application used to generate the backup media. The component parts of a file may be read from a medium, including content and metadata pertaining to a file. These pieces of content and metadata may then be stored and associated. To avoid duplication of data, pieces of content and metadata may be compared to previously stored content and metadata. Furthermore, using these same methods and systems the content and metadata of a file may be associated with a location where the file resided. A database which stores these components and allows linking between the various stored components may be particularly useful in implementing embodiments of these methods and systems.