Physical Extent Manager for Scalable Storage Pool
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems, such as Data Domain, require large and complex disk farms to meet capacity and performance needs, and lack flexibility in scaling and managing storage across both physical and virtual environments.
Innovation Solution
A method for managing extents in a system with a protection pool, utilizing a Physical Extent Manager (PEM) that collects metadata, creates request queues, and listens for multicast messages to dynamically allocate and migrate physical extents across nodes, allowing for incremental scaling and resilience against node failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a scale up architecture with single node appliances is used, then the system is simple to deploy, but it requires very large and complex disk farms to meet capacity and performance requirements
Solution Approach 1:
The system is divided into multiple independent storage nodes that can be individually managed and scaled. Each node maintains its own disk farms, and the system presents a unified namespace across all nodes. This segmentation allows capacity and performance to be increased by adding nodes rather than expanding individual disk farms, thereby reducing the complexity of each disk farm while meeting overall capacity requirements.
Solution Approach 2:
The architecture transitions from a single-dimension scale-up model (adding more disks to one system) to a multi-dimension model (adding nodes across the network). By introducing the network dimension and distributing storage across multiple nodes, the system can meet capacity requirements without proportionally increasing the complexity of individual disk farms.
2Productivity
If protection pool architecture with multiple nodes is used, then linear scaling of performance and capacity is achieved, but the system becomes more complex to manage
Solution Approach 1:
Each storage node is designed to be universally functional, capable of performing all storage operations independently. The nodes share a common namespace and can access each other's storage resources through the protection pool architecture. This multi-functionality allows linear scaling of performance and capacity while maintaining consistent management interfaces, as each node can operate autonomously yet contribute to the unified system.
Solution Approach 2:
The patent introduces a namespace service and metadata management layer that acts as an intermediary between clients and the distributed storage nodes. This intermediary handles the complexity of locating, managing, and coordinating resources across multiple nodes, presenting a simplified interface to users while enabling linear scaling of the underlying infrastructure.
3Reliability
If data is distributed across multiple nodes, then resilience against node failures is improved, but the complexity of maintaining storage access during failures increases
Solution Approach 1:
The patent merges the namespace management and data location information into a centralized metadata service that is replicated across nodes. When a node fails, the system automatically retrieves updated namespace information from other nodes and redirects access accordingly. This merging of namespace and data management simplifies failure handling by providing automatic failover without requiring complex manual intervention.
Solution Approach 2:
The system implements continuous feedback mechanisms where nodes periodically exchange status information and the namespace service monitors node availability. When a failure is detected, the system automatically updates metadata and redirects client access to healthy nodes. This feedback-driven approach maintains reliability while managing failure complexity through automated responses rather than manual processes.
Data Source
AI summary
A method of managing extents of a file system having a protection pool includes collecting and initializing physical extent manager (PEM) metadata, using a PEM daemon thread. The PEM is configured to run on each of a number of nodes. The method also includes creating a request queue, using the PEM daemon thread, for all requests submitted to the PEM. The method also includes scanning the request queue, using a PEM worker thread, to handle incoming requests submitted to the PEM. The method also includes listening for multicast messages, using a PEM multicast listener thread, to be handled by the PEM worker thread.