Distributed Curator for Virtual Storage Maintenance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional virtualization technologies face inefficiencies in managing and optimizing storage operations, particularly in handling maintenance and optimization tasks such as garbage collection, due to a lack of scalable solutions for advanced metadata techniques.
Innovation Solution
A distributed curator entity manages the virtual storage system, utilizing MapReduce operations for full and partial scans to oversee the system's health and performance, allowing for efficient execution of maintenance and optimization tasks, including garbage collection, replication, and data migration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional virtualization technologies are used to manage storage operations, then basic storage functionality is provided, but maintenance and optimization tasks such as garbage collection cannot be efficiently performed due to lack of scalable solutions for advanced metadata techniques
Solution Approach 1:
The storage system is segmented into multiple metadata servers that independently manage different portions of the metadata namespace. Each metadata server handles a specific subset of metadata operations, allowing maintenance tasks like garbage collection to be distributed and executed in parallel across multiple servers, thereby improving efficiency without requiring complex centralized coordination
Solution Approach 2:
Metadata servers act as intermediary components between the virtualization layer and the underlying storage infrastructure. These intermediaries provide scalable metadata management capabilities that enable efficient execution of maintenance tasks while abstracting the complexity from both the virtualization layer and the storage hardware
2Ease of operation
If coarse-grained volume management is used in storage servers, then storage administration is simplified, but granular administrative tasks and storage optimizations cannot be performed on specific regions or files
Solution Approach 1:
The metadata namespace is segmented into multiple independent zones or partitions, each managed by a dedicated metadata server. This segmentation enables administrators to apply different policies and optimizations to specific regions or files while maintaining simplified overall management through the modular architecture
Solution Approach 2:
Different portions of the storage system can have different metadata management characteristics tailored to their specific needs. Each metadata server can be configured with optimal settings for its designated namespace portion, allowing granular control and optimization while maintaining ease of operation through standardized interfaces
3Productivity
If networked storage devices are shared among multiple VMs, then resource utilization improves, but direct attached storage and certain storage optimizations cannot be utilized
Solution Approach 1:
The metadata server architecture provides universal metadata management capabilities that work across different storage architectures. The same metadata servers can manage both networked shared storage and direct-attached storage, enabling resource utilization benefits of sharing while maintaining the flexibility to utilize storage optimizations specific to different storage types
Data Source
AI summary
Disclosed is an approach for overseeing the overall state of a virtual storage system, and for taking actions as necessary to manage the health and efficient performance of that system. A curator entity is provided on a distributed basis to manage and perform these functions. A master curator manages this workload that is performed by multiple slave curators. MapReduce operations are performed to implement the curator workload. Periodically, the master curator coordinates a full scan of all the metadata in the system using three MapReduce operations. Partial scans may also be run using a single MapReduce operation.