Data Chunking and Identifier Generation for Backup Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup methods inefficiently store repeated data, leading to wasteful use of storage medium when full backups are performed, as they copy the same data multiple times.
Innovation Solution
A system that chunks data into unique identifiers and transfers only new or unknown data chunks to a remote secondary storage device, using a chunking and identifier generation module to process data into manageable sections and send only necessary data, reducing redundant transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full backups are performed regularly, then data recovery reliability is improved, but storage medium efficiency deteriorates due to redundant data copying
Solution Approach 1:
The patent segments data into fixed-size chunks with unique identifiers (e.g., 4KB chunks with 64-bit IDs). This segmentation allows the backup system to treat individual chunks as independent units, enabling selective transmission of only new or modified chunks rather than redundant full backups, thus improving storage efficiency while maintaining complete data backup capability
Solution Approach 2:
The patent uses chunk identifiers as digital fingerprints to detect duplicate data. By comparing chunk IDs between primary and secondary storage, the system identifies identical data blocks and avoids copying them again. This copying principle with identification enables incremental backups that maintain data completeness while eliminating redundant storage operations
2Loss of substance
If incremental backups are performed, then storage efficiency is improved, but network traffic increases due to frequent data transfers
Solution Approach 1:
The patent pre-calculates and stores chunk identifiers for all data blocks before actual backup operations. This preliminary indexing allows the system to quickly identify which chunks need transfer by simple ID comparison rather than transferring data for verification, significantly reducing network traffic while maintaining incremental backup efficiency
Solution Approach 2:
The patent implements a feedback mechanism where the secondary storage device sends back information about existing chunk IDs to the primary device. This feedback loop enables the primary device to determine exactly which chunks are new or modified and need transmission, optimizing network traffic by transferring only necessary data while maintaining storage efficiency
Data Source
AI summary
An apparatus, for connection to a storage device, comprising: a module for communication with a software application and operable to receive instructions to copy data to the storage device; a chunking and identifier generation module operable to receive the data, to process the data into chunks, to generate a first chunk identifier, representative of the identity of a first of the chunks of data and, upon processing of the data, to initiate the issuance of a confirmation signal to the software application indicating that the data has been copied to the storage device; and an interface for communication with the storage device, wherein the interface is operable to send the first chunk identifier to the storage device, and to send the first chunk of data to the storage device upon receipt of a transfer instruction from the storage device.