Cloud File Deduplication via Multi-Signature Reference Lists
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based backup solutions face challenges in identifying and avoiding redundant file transfers across devices with different operating systems, leading to erroneous duplication due to transcoding and byte-wise differences in file data, especially when files are transcoded for compatibility.
Innovation Solution
A content management system uses a duplication reference list that stores multiple signature types, including hash-based signatures and platform-specific identifiers, to accurately identify and prevent the transfer of duplicate files by comparing signatures of newly discovered content items with previously stored files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cloud-based backup services perform routine backups of new or altered data, then data backup reliability is improved, but redundant file transfers increase leading to storage waste
Solution Approach 1:
The system performs preliminary identification of duplicate files by comparing hash signatures before initiating backup transfers. The cloud service generates a hash of each local file, checks it against previously backed-up file hashes, and only transfers files that are confirmed to be new or modified, thereby preventing redundant transfers in advance
Solution Approach 2:
The system uses hash signatures as digital copies or representations of file content to identify duplicates without transferring the actual files. By comparing these lightweight hash copies against a database of existing file hashes, the system can determine whether full file transfers are necessary
2Adaptability or versatility
If files are transcoded for compatibility across different operating systems, then system adaptability is improved, but file data changes causing erroneous duplication
Solution Approach 1:
The system performs preliminary hashing of files before transcoding operations. By generating and storing the original file hash before any format conversion, the system can later verify whether a transcoded file represents a genuine modification or merely a format change, preventing erroneous duplication
Solution Approach 2:
The hash signature acts as an intermediary that bridges the original file and its transcoded versions. The hash provides a reference point to determine whether changes in file data are due to transcoding or actual content modifications, allowing the system to distinguish between format variations and genuine updates
3Measurement precision
If multiple signature types are used to identify file duplicates, then duplicate detection accuracy is improved, but system complexity increases
Solution Approach 1:
The system segments the duplicate detection process into multiple independent signature checks. Instead of using a single complex identification method, it employs separate hash-based signatures and platform-specific identifiers that can be evaluated independently, improving accuracy while maintaining manageable system complexity through modular processing
Data Source
AI summary
Aspects of the subject technology relate to ways to avoid file duplication in a cloud storage service. In some aspects, a method of the technology includes steps for retrieving, via a computer network, one or more files from a first memory location, identifying a first signature and a second signature for each of the files, and storing the first signature and the second signature into a duplication reference list. In some aspects, the method further includes steps for analyzing files in a second memory location to identify duplicate files using the duplication reference list. Systems and computer-readable media are also provided.