Parallel Backup System Using Identity Representation Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data archival and backup systems face inefficiencies in reducing network traffic and storage queries, as they often send unnecessary data for backup, leading to increased network load and storage requirements.
Innovation Solution
A backup system that uses an identifier to generate a unique identity representation of data objects, such as through hash functions, to determine if data is already stored, thereby minimizing network traffic by only transmitting unique data objects and allowing multiple backup stores to service a computing environment in parallel.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional backup systems send all data objects to a central backup server for storage, then all data can be archived, but network traffic and storage queries increase significantly
Solution Approach 1:
The system generates identity representations (hashes) of data objects and performs preliminary comparison against existing backup store contents before initiating actual data transmission. This preliminary action identifies duplicate data objects, allowing the system to skip transmitting data that already exists in the backup store, thereby reducing network traffic while ensuring all unique data is archived
Solution Approach 2:
The system extracts only the essential identifying feature of each data object (its identity representation or hash) for comparison purposes, rather than transmitting or storing the entire data object. This extraction allows the system to determine duplication status using minimal data, reducing both network traffic during backup operations and storage requirements
2Reliability
If conventional backup systems send all data objects to a central backup server, then complete backup is achieved, but the number of storage queries increases
Solution Approach 1:
The system extracts and compares only identity representations (hash values) rather than full data objects when determining whether to store data. This extraction approach reduces storage query complexity and improves efficiency by comparing compact identifiers instead of large data sets, while still ensuring complete backup of unique data objects
Solution Approach 2:
The system performs preliminary comparison of identity representations before initiating storage operations. This preliminary action filters out duplicate data objects that already exist in the backup store, reducing the number of actual storage queries needed while maintaining backup completeness for all unique data
3Productivity
If multiple backup stores are used to service a computing environment in parallel, then backup capacity and speed increase, but determining which store receives which data becomes complex
Solution Approach 1:
The system assigns specific identity representation value ranges to different backup stores, creating local responsibility zones. Each backup store is responsible for a specific subset of identity representations, allowing parallel processing across multiple stores while simplifying routing decisions. The identifier determines the appropriate backup store based on the identity representation value, reducing overall system complexity through localized management
4Loss of energy
If identity representations are used to determine backup necessity, then network traffic is reduced, but generating and comparing identity representations requires processing overhead
Solution Approach 1:
The system uses identity representations (hash values) as disposable, lightweight substitutes for actual data objects during comparison operations. These cheap, compact identifiers can be generated and compared with minimal processing overhead compared to handling full data objects, reducing network traffic while the processing cost remains acceptable due to the efficient nature of hash function computation
Data Source
AI summary
A backup system can be provided, which system can comprise an identifier operable to identify a data object for possible backup and a generator operable to generate an identity representation for the data object on the basis of the content thereof. The identity representation can be transmitted to a backup store where it can be tested against identity representations for previously stored objects. The system can also comprise a receiver operable to receive from the backup store an indication of whether the data object described by the identity representation has previously been stored at the backup store. The system can be operable to transmit the data object to the backup store in the event that the data object described by the identity representation has not previously been stored at the backup store.