Duplicate Fork Detection in Archiving Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current archiving software fails to efficiently detect and store identical or nearly identical files, leading to redundant compression and increased storage requirements, as it relies on file ordering and compression history, which limits the detection of duplicate files and forks.
Innovation Solution
A method that detects identical or nearly identical files by sorting and comparing forks, using direct comparison or hashing algorithms, and stores references to the first compressed copy, eliminating the need for repeated compression and allowing efficient storage of duplicate data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If current archiving software compresses each data file as a whole without detecting duplicates, then the compression process is simple to implement, but storage space is wasted and compression efficiency is reduced
Solution Approach 1:
The patent applies preliminary action by sorting files by attributes (size, type, name) before compression to group potential duplicates together. This preliminary organization enables efficient duplicate detection without requiring complex mechanisms during the compression phase itself, as duplicates are already positioned adjacent to each other for easy identification.
Solution Approach 2:
The patent uses copying by creating references to the first occurrence of duplicate files rather than storing complete copies. When a duplicate is detected, the system stores a reference pointing to the original file's compressed data, eliminating redundant storage while maintaining the ability to retrieve all file versions through the reference mechanism.
2Quantity of substance
If solid archive mode concatenates and compresses files as a single large block, then nearby identical data can be detected, but files too remote from each other cannot reference each other's data
Solution Approach 1:
The patent applies segmentation by dividing the archive into separate compressed blocks for each file rather than creating one large concatenated block. This allows the duplicate detection mechanism to compare files independently across the entire archive, enabling references between files regardless of their positional distance, while still maintaining compression efficiency for detected duplicates.
3Productivity
If duplicate detection is performed by sorting files by attributes, then identical files are ordered close to each other, but the detection is still dependent on window size and compression history
Solution Approach 1:
The patent uses copying to store reference information that points to the location of the first occurrence of duplicate files. This reference copying mechanism eliminates the need for complex window size calculations and compression history tracking, as the reference directly identifies the original file regardless of positional relationships or compression parameters.
4Productivity
If the second file is compressed even when identical to the first, then compression processing occurs, but storage space is wasted on redundant data
Solution Approach 1:
The patent applies copying by storing references to the first occurrence of duplicate files instead of compressing and storing complete copies of subsequent duplicates. This reference copying approach eliminates redundant storage while maintaining minimal processing, as the system only needs to create and store reference information rather than performing full compression operations on duplicate files.
Data Source
AI summary
A method to efficiently detect, store, modify, and recreate fully or partially duplicate file forks is described. During archive creation or modification, sets of fully or partially duplicate forks are detected and a reduced number of transformed forks or fork segments are stored. During archive expansion, one or more forks are recreated from each full or partial copy.