Universal Archive Volume Format for Efficient Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing archive volume formats complicate deduplication processes due to the intermixing of metadata, headers, and file data, leading to inefficiencies and vendor-specific solutions, which are not universally compatible with deduplication algorithms in filesystems and storage systems.
Innovation Solution
A universal archive volume format is created, separating metadata and file data into distinct files, aligning file data for optimal deduplication, and allowing user-configurable options for block sizes and padding to ensure compatibility with various deduplication engines, thereby facilitating efficient deduplication across different storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If metadata and file data are mixed together in a single archive file, then the archive structure is simple and easy to manage, but deduplication efficiency deteriorates because the file data becomes fragmented and unique
Solution Approach 1:
The patent divides the archive file into two separate files: a metadata file containing all metadata and headers, and an aligned data file containing only file data. This segmentation allows deduplication algorithms to work efficiently on the aligned data file while the metadata file remains separately manageable, resolving the contradiction between structural simplicity and deduplication efficiency.
Solution Approach 2:
The patent extracts file data from the mixed archive structure and places it in a separate aligned data file. This extraction removes the harmful mixing of metadata and data, enabling deduplication algorithms to process pure file data without interference from metadata, thereby improving deduplication efficiency while maintaining manageable archive structures.
2Adaptability or versatility
If vendor-specific deduplication algorithms are implemented in archive programs, then deduplication can be performed on archived data, but compatibility with universal deduplication systems deteriorates and inefficiencies arise from duplicate deduplication
Solution Approach 1:
The patent creates a universal archive format where the aligned data file contains file data in a standardized structure that can be deduplicated by any deduplication system, whether built into filesystems, storage systems, or third-party tools. This universality eliminates the need for vendor-specific algorithms and prevents duplicate deduplication, improving efficiency while maintaining broad compatibility.
Solution Approach 2:
The aligned data file acts as an intermediary between the archive program and deduplication systems. By providing file data in a standardized, alignment-based format, it enables seamless interaction with various deduplication engines without requiring vendor-specific processing, thus improving both compatibility and efficiency.
3Quantity of substance
If file data is broken into small chunks and compacted with metadata, then transmission and storage efficiency improve, but deduplication capability deteriorates because the original file structure is lost
Solution Approach 1:
The patent performs preliminary alignment of file data before storing it in the aligned data file. By organizing data in fixed-size blocks with proper alignment markers beforehand, it enables deduplication algorithms to efficiently identify and eliminate duplicates while maintaining the compact, chunked structure for efficient storage and transmission.
Solution Approach 2:
The patent changes the organizational parameters of file data by introducing fixed block sizes, alignment markers, and sequential block numbering. These parameter changes transform fragmented chunks into a structured format that maintains high data density while enabling effective deduplication through consistent block boundaries and alignment.
Data Source
AI summary
A method for creating a volume that contains data from an original stream of multiple files, and which can be optimally deduplicated by an underlying deduplication storage system. The method comprises receiving data records representing metadata and file data, at least a part of which are already separated, separating the metadata and the file data into a first file and a second file, the first file and the second file being paired, the first file called Metadata Volume containing metadata, header data and references to the file data, and the second file called Aligned Volume containing file data only. A further part of the records which contain both metadata and file data are separated into metadata and file data and then subjected to the step of separating the metadata and the file data into the first file and the second file.