Metadata Distribution in Storage Arrays for Multi-Disk Failure Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems, such as RAID systems, face challenges in data recovery when multiple disks fail, as they often rely on redundancy mechanisms that are insufficient for handling extensive failures, leading to data loss.
Innovation Solution
A data storage array with a policy-driven storage manager that uses metadata to determine data storage policies across multiple devices, ensuring that critical data is duplicated and distributed across all devices, allowing for recovery even if multiple devices fail, by employing a file management system and storage manager to manage virtual and physical blocks and apply policies for redundancy and data placement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional RAID redundancy mechanisms are used, then data can be recovered if one or two disks fail, but data recovery capability deteriorates when multiple disks fail
Solution Approach 1:
The patent segments data into virtual blocks that can be independently distributed across multiple physical storage devices. Each virtual block can be placed on different devices based on policies, allowing the system to handle multiple failures by recovering from any combination of failed devices. This segmentation enables flexible data distribution that traditional RAID cannot achieve.
Solution Approach 2:
The patent introduces a virtual block layer between the file system and physical storage devices, adding a dimensional abstraction that enables policy-driven data placement. This virtual layer allows data to be distributed across devices in a way that traditional RAID stripe/parity mechanisms cannot, providing resilience against multiple failures through policy-based decisions rather than fixed RAID configurations.
2Reliability
If array metadata is stored on every device, then data can be recovered if devices fail, but storage space is consumed on all devices
Solution Approach 1:
The patent applies partial redundancy by storing array metadata on every device only for critical virtual blocks that require it, rather than duplicating all data. The policy engine determines which virtual blocks need metadata replication based on their importance and failure scenarios, applying redundancy only where necessary to achieve recoverability without consuming excessive storage space.
Solution Approach 2:
The patent implements local quality by applying different metadata storage policies to different virtual blocks based on their characteristics and requirements. Not all blocks require the same level of metadata redundancy - the policy engine evaluates each block and applies metadata storage only where needed, optimizing the balance between recoverability and storage consumption.
3Reliability
If data is distributed across multiple devices, then system reliability improves, but device complexity increases
Solution Approach 1:
The patent introduces a storage manager as an intermediary component that handles the complexity of data distribution across multiple devices. The storage manager receives data from the file system, applies policies to determine optimal placement, and manages metadata storage on individual devices. This intermediary abstraction hides the complexity of multi-device management from the file system and applications, maintaining high reliability while managing complexity centrally.
Solution Approach 2:
The policy engine enables the storage system to make autonomous decisions about data placement and metadata storage based on predefined policies. The system self-determines which blocks to replicate, where to place them, and how to manage redundancy without requiring complex manual configuration or external control, reducing operational complexity while maintaining distributed reliability.
Data Source
AI summary
A data storage array may be made up of several storage devices, each of which may contain array metadata that may allow portions of the storage array to be used. A system may have a file system manager that may receive and respond to file system commands and a storage device manager that may store data on the several storage devices. Array metadata defining where data is stored within the storage array is stored on each device within the array. A policy engine may identify data to be stored on the array and determine if the data contains array metadata or other types of data and may store the data on every device, devices having specific characteristics, two or more devices, or one device.