Object Storage Metadata Migration Using Supplemental Data Structures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Metadata stored in object storage is susceptible to deletion and/or modification during data migration from one container to another, especially when migrating between different cloud providers, leading to data loss and integrity issues, particularly when migrating to file storage systems.
Innovation Solution
A novel data migration strategy that involves directly injecting object storage metadata into file storage properties, using a supplemental data structure to preserve metadata integrity by maintaining its native form and avoiding direct access to the object data, allowing for targeted and secure migration of specific metadata portions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional data migration is used to move objects from one container to another, then data transfer is achieved, but metadata is lost or modified during migration
Solution Approach 1:
The system performs preliminary actions by generating a supplemental data structure containing metadata array and source identifier before the actual data migration occurs. This pre-prepared metadata package is then injected into the target container, ensuring metadata integrity is maintained throughout the migration process without requiring post-migration reconstruction.
Solution Approach 2:
The supplemental data structure acts as an intermediary carrier that transports metadata from the source container to the target container. This intermediate structure includes the metadata array and source identifier, serving as a bridge that preserves metadata integrity during the migration process between different cloud storage ecosystems.
2Loss of information
If metadata is accessed and processed during migration, then metadata can be transferred, but computing resources are consumed and data security risks increase
Solution Approach 1:
The system extracts only the necessary metadata components (metadata array and source identifier) into a separate supplemental data structure, leaving the main object data untouched. This extraction approach allows metadata to be transferred and preserved without requiring access to or processing of the actual object data, minimizing computing resource consumption and enhancing security.
Solution Approach 2:
The migration process is segmented into distinct components: the main object data and the supplemental metadata structure. By separating metadata transfer from object data migration, the system can preserve metadata integrity while avoiding the computational overhead and security risks associated with accessing and processing the actual object data during migration.
3Loss of information
If metadata is reconstructed after migration, then data completeness is achieved, but time and resources are wasted
Solution Approach 1:
The system performs the metadata preservation action preliminarily by embedding the complete metadata array and source identifier into the supplemental data structure before migration. This eliminates the need for post-migration reconstruction, as the metadata is already in its final, complete form upon arrival at the target container, saving significant time and computational resources.
Data Source
AI summary
Systems and methods for novel uses and/or improvements to data migration. In particular, systems and methods for data migration of metadata stored in object storage from one container to another, especially in instances when the metadata is destined for ingestion by an artificial intelligence application. The systems and methods ensure that all metadata (e.g., metadata stored in object storage) is preserved during data migration, including metadata such as content type, object lock mode, object lock retain until date, and/or custom metadata.