Distributed Storage Metadata Management for Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in managing high-capacity, scalable storage for rich digital media and unstructured business content due to high costs associated with specialized hardware, complexity in tiered storage, and difficulties in maintaining system reliability and accessibility as data grows.
Innovation Solution
A data storage system that manages files by applying policies based on predetermined conditions, using a resource manager to determine available resources, and employing a metadata system with layout storage objects to handle synchronous and asynchronous mirrors, ensuring data availability and reliability across multiple storage locations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If specialized hardware platforms are used to achieve required levels of performance and reliability, then storage performance and reliability are improved, but customer prices increase and volume economics are not supported
Solution Approach 1:
The patent employs inexpensive, commodity hardware components (standard disk drives, off-the-shelf servers) rather than specialized, expensive hardware. The system achieves reliability through software-based redundancy and distributed architecture, allowing the use of cheaper, easily replaceable components that can be rapidly provisioned and deprovisioned.
Solution Approach 2:
The system changes the fundamental parameters of storage architecture by moving from centralized, specialized hardware to distributed, commodity hardware. It transforms the reliability model from hardware-based redundancy to software-based distributed replication across multiple nodes, enabling cost-effective scaling while maintaining reliability.
2Ease of manufacture
If tiered storage is implemented to reduce capital and operational costs, then storage costs are reduced, but system complexity increases and management becomes strained
Solution Approach 1:
The patent implements a unified storage architecture where a single distributed system provides multiple functions (storage, replication, load balancing, failure recovery) rather than requiring separate tiered storage systems. This universal approach eliminates the complexity of managing multiple storage tiers while providing cost-effective scaling through software-defined policies.
Solution Approach 2:
The system merges storage management, replication control, and resource allocation into a single integrated software platform. By combining these functions into one system rather than separate tiered storage layers, it reduces operational complexity while maintaining cost efficiency through centralized policy management.
3Reliability
If multiple copies of information sets are created on each storage tier, then data durability and accessibility are improved, but data replica incoherence occurs and management becomes difficult
Solution Approach 1:
The patent implements centralized coordination with feedback mechanisms that track the state of all data replicas across the distributed system. The control plane monitors replication status, detects inconsistencies, and coordinates recovery operations, ensuring data coherence while maintaining durability through continuous feedback from distributed nodes.
Solution Approach 2:
The system introduces a centralized control plane as an intermediary between distributed storage nodes. This mediator coordinates replication operations, manages consistency protocols, and resolves conflicts between multiple copies, enabling durable storage without the management overhead of direct peer-to-peer replica coordination.
4Quantity of substance
If the system scale is increased to handle large capacity requirements, then storage capacity is improved, but difficulty in maintaining system and managing environment increases
Solution Approach 1:
The patent implements self-managing distributed storage nodes that automatically perform health monitoring, failure detection, data replication, and recovery operations. Each node autonomously manages its own state while contributing to the overall system, eliminating the need for manual intervention as capacity scales and reducing operational complexity.
Solution Approach 2:
The system employs dynamic resource allocation and automatic scaling capabilities that adapt to changing capacity requirements. The distributed architecture allows nodes to be dynamically added or removed without manual reconfiguration, and the system automatically rebalances data and updates replication policies, making large-scale management effortless.
Data Source
Figure 1
Figure 2A~2B
Figure 3
AI summary
There is provided a method of managing a plurality of files in a storage system, comprising: providing at least one metadata location server of the storage system, wherein the storage system has a plurality of interconnected independent servers and wherein the at least one metadata location server has a table with plurality of file identifiers, each of the file identifiers including a pointer to a metadata file object stored on a metadata server and containing a layout storage object that indicates storage parameters for corresponding files; and accessing a particular one of the files by obtaining information from the at least one metadata location server to access the metadata server containing the layout storage object corresponding to the particular file to determine synchronous and asynchronous mirrors for the file.