Intra-cluster Migration for Elastic Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Elastic Cloud Storage (ECS) systems face inefficiencies in data migration due to operating at a chunk level, requiring repeated replication of entire data sets and complete rebuilds of data tables, leading to performance issues and garbage collection challenges during hardware upgrades.
Innovation Solution
Implementing intra-cluster migration (ICM) at the chunk level, which preserves chunk IDs and eliminates repeated replication by moving data chunks between geographically distributed storage devices, updating storage reference data structures, and ensuring adequate capacity and consistency in target storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data migration is performed at the object level in ECS, then data can be moved between storage devices, but repeated replication of the entire data set is necessary and complete rebuild of Data Tables is required
Solution Approach 1:
The patent segments data at the chunk level rather than object level. Each object is divided into multiple chunks that can be independently migrated. This segmentation allows only the necessary chunks to be moved during migration, eliminating the need to replicate entire data sets or rebuild complete Data Tables, thus resolving the efficiency problem while maintaining migration capability.
2Loss of information
If complete rebuild of Data Table is performed during migration, then data location information can be updated, but large amount of garbage is generated that needs collection
Solution Approach 1:
The patent performs preliminary actions by maintaining Data Table consistency throughout the migration process. Chunk location information is updated incrementally as chunks are migrated, rather than performing a complete rebuild afterward. This preliminary updating approach ensures data location accuracy is maintained while avoiding the generation of large amounts of garbage data that would require collection.
3Adaptability or versatility
If geographically distributed storage devices are used, then system scalability is improved, but migration complexity increases
Solution Approach 1:
The patent implements a universal chunk-level migration mechanism that works across geographically distributed storage devices. The same migration protocols and data structures (chunks, coding chunks, Data Tables) are used regardless of location, providing multi-functionality that handles both local and remote migrations uniformly. This universality enables system scalability while keeping migration complexity manageable through standardized procedures.
Data Source
AI summary
Facilitating intra-cluster migration of data in an elastic cloud storage environment is provided herein. A system can comprise a processor and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations. The operations can comprise scheduling a migration of a data chunk from a source storage node to a target storage node. Further, the operations can comprise facilitating the migration of the data chunk from a first location in the source storage node to a second location in the target storage node. Data indicative of an identity of the data chunk can be migrated with the data chunk. The operations can also comprise replacing the first location with the second location in a storage mapping table.