Container Data Migration with CSI Snapshots and Changed-Block Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems lack efficient mechanisms for seamless migration and management of data across diverse environments, including on-premises and public cloud setups, while ensuring data integrity and compliance with security and compliance policies, particularly in containerized applications orchestrated by frameworks like Kubernetes.
Innovation Solution
A distributed data storage system with a proprietary Container Storage Interface (CSI) driver and a container data mover that enables efficient data migration between storage clusters, using metadata-based snapshots and change block tracking, along with a barrier logic to ensure controlled operations, while providing snapshot scheduling and retention policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data migration methods are used between storage systems, then data can be migrated, but the migration process is time-consuming and disrupts application operations
Solution Approach 1:
The system creates a snapshot of the source storage volume before migration begins. This snapshot captures the complete state of the volume including all data and metadata, allowing the destination storage system to be pre-configured with this information before actual data transfer occurs. This preliminary action enables faster migration by avoiding the need to transfer and re-process all data from scratch.
Solution Approach 2:
The system uses kernel-to-kernel copying to replicate payload data directly between storage systems at the kernel level, bypassing higher-level processing layers. This copying mechanism is complemented by metadata-based change block tracking that identifies only the changed blocks between source and destination, enabling efficient incremental migration of only necessary data portions.
2Reliability
If comprehensive data migration is performed, then all data is transferred, but metadata operations slow down the migration process
Solution Approach 1:
The system segments the data migration process into separate operations for payload data and metadata. Payload data is transferred using kernel-to-kernel copying for high speed, while metadata is handled through separate metadata operations. This segmentation allows each component to be optimized independently, with payload data migration benefiting from high-speed copying while metadata operations are handled efficiently through change block tracking.
Solution Approach 2:
The system implements metadata-based change block tracking that continuously monitors and identifies changed blocks between source and destination storage systems. This feedback mechanism allows the migration process to adapt dynamically, transferring only the changed blocks rather than all data, thereby improving migration efficiency while maintaining complete data integrity through accurate change detection.
3Productivity
If data migration is performed without controlled progression, then migration is faster, but data integrity and consistency cannot be ensured
Solution Approach 1:
The system performs preliminary actions by creating a complete snapshot of the source storage volume before migration begins. This snapshot serves as a reference point that ensures data consistency throughout the migration process. The barrier logic uses this preliminary snapshot to verify that all data and metadata are properly accounted for before finalizing the migration, ensuring integrity without slowing down the actual data transfer.
4Adaptability or versatility
If storage systems are not integrated with application orchestrators, then system independence is maintained, but seamless migration and management across diverse environments is difficult
Solution Approach 1:
The system implements universal integration with application orchestrators like Kubernetes through a standardized CSI driver interface. This universal interface allows the same storage system to operate seamlessly across different environments (on-premises, public cloud, hybrid) and with different orchestrators without requiring environment-specific custom integration logic, thereby improving adaptability while managing complexity through standardization.
Data Source
AI summary
A distributed data storage system comprises features for integration with application orchestrators such as Kubernetes, and includes a proprietary Container Storage Interface (CSI) driver. Features include setting snapshot scheduling and retention policies, and a “container data mover” that replicates data from a source to a distinct destination distributed data storage system. The migration may be configured one-to-one, one-to-many, unidirectional, and/or bi-directional. Metadata-based snapshots and metadata-based changed block tracking identify payload data to move from source to destination within application orchestrator frameworks at both ends. Payload data is migrated from source to destination using different techniques than those used for migrating metadata, e.g., kernel-to-kernel for copying payload data versus ordinary metadata writes. An illustrative barrier logic ensures that the migration follows a controlled progression of operations.