File System Archiving With Checksums for Cloud Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems lack efficient solutions for archiving file system data, replicating archives to cloud storage, and verifying, extracting, and updating files from these archives.
Innovation Solution
A method and system for generating an archive data structure that includes a summary file, manifest file, and archive data files, utilizing checksums and metadata to ensure data integrity, enabling efficient packing, verification, extraction, and update of large file systems across networked storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional storage systems are used for archiving large file systems, then data storage capability is provided, but efficiency and reliability of archiving, replicating, verifying, extracting and updating files is insufficient
Solution Approach 1:
The patent segments the archiving process into distinct phases: creating archive data structures with metadata, replicating to cloud storage, verifying integrity through checksums, extracting files, and updating archives. This segmentation allows each operation to be optimized independently, improving overall productivity while maintaining reliability through structured verification steps.
Solution Approach 2:
The patent performs preliminary actions by creating comprehensive metadata and checksums before archiving, and by verifying data integrity before extraction. These preliminary verification steps ensure data integrity is established in advance, preventing issues during migration and extraction operations.
2Adaptability or versatility
If large file systems are archived to cloud storage, then data migration capability is achieved, but time consumption for archiving and verification increases
Solution Approach 1:
The patent creates archive data structures with embedded metadata and checksums as preliminary actions before migration. This allows verification operations to be performed efficiently during and after extraction without requiring separate verification passes, reducing overall time consumption while maintaining the capability to handle large file systems.
Solution Approach 2:
The patent implements feedback mechanisms through checksum verification that provides immediate confirmation of data integrity during the extraction process. This feedback allows the system to identify and correct issues quickly, reducing the overall time required for verification and improving the efficiency of large file system migration.
Data Source
AI summary
Methods and systems for a networked storage environment are provided. One method includes generating a first and a second batch of entries in response to scanning a source file system, each batch of entries associated with one or more directories of the source file system and indicating a path to a file associated with each entry; determining, by a first worker process, a first checksum for data associated with the first batch of entries loaded in a first buffer; appending, the first buffer contents processed by the first worker process to a first archive file; and generating an archive data structure having a manifest file storing metadata for the first batch and a second batch entries with a first checksum determined by the first worker process and a second checksum determined by a second worker process, and data from the first archive file and a second archive file.