Metadata Server Segmentation for Scalable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in managing high-capacity, scalable storage for rich digital media and unstructured business content, including high costs due to specialized hardware, complexity in tiered storage, and difficulties in maintaining system reliability and data coherence as capacity grows.
Innovation Solution
A data storage system utilizing multiple metadata servers, each managing a subset of file objects, with a metadata location server determining which server contains specific file data, and implementing a layout storage object tree for mirroring and resource management to optimize storage and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If specialized hardware platforms are used to achieve required levels of performance and reliability, then system performance and reliability are improved, but customer prices increase and volume economics are not supported
Solution Approach 1:
The system segments metadata management across multiple independent metadata servers, each handling a subset of file objects. This allows the system to scale using commodity hardware rather than requiring specialized hardware platforms, as each server can be a standard off-the-shelf machine. The segmentation enables linear scalability while maintaining reliability through distributed architecture.
Solution Approach 2:
The patent employs universal metadata servers that can handle multiple functions including storing metadata, tracking leases, managing file object information, and coordinating access. These servers run on standard hardware platforms and provide multi-functional capabilities that replace the need for specialized hardware, achieving cost-effectiveness while maintaining system reliability.
2Ease of manufacture
If tiered storage is implemented to reduce capital and operational costs, then storage costs are reduced, but system complexity increases and data replica incoherence occurs
Solution Approach 1:
The patent merges multiple storage tiers into a unified object storage system where all data is accessed through a common interface and metadata management layer. The metadata servers maintain a unified view of all file objects regardless of physical storage location, eliminating the complexity of managing separate tiered systems while still allowing data to reside on cost-effective commodity hardware.
Solution Approach 2:
The metadata servers act as intermediaries between clients and physical storage devices. They manage the mapping between logical file objects and physical storage locations, handling lease coordination and access control. This intermediary layer simplifies the system architecture by abstracting away the complexity of tiered storage management while maintaining cost benefits.
3Reliability
If multiple copies of information sets are created across storage tiers, then data availability and performance are improved, but data replica incoherence and management complexity increase
Solution Approach 1:
The metadata servers implement lease-based feedback mechanisms where clients must acquire leases before accessing file objects. The system tracks lease states and coordinates access across multiple replicas, providing real-time feedback about data coherence and access permissions. This feedback mechanism ensures data consistency while allowing multiple copies to exist for availability and performance.
Solution Approach 2:
The patent changes the state parameters of file objects to track their replication and access status. By using leases as a state parameter that indicates which client has permission to access or modify a file object, the system can manage multiple replicas coherently. The lease state acts as a control parameter that coordinates access across distributed copies without requiring complex replication management.
4Quantity of substance
If the system scales to accommodate billions of file objects, then storage capacity is improved, but metadata management complexity and failure handling difficulty increase
Solution Approach 1:
The patent segments the metadata management space by dividing file objects into subsets, with each metadata server responsible for a specific range or category of file objects. This segmentation allows the system to scale to billions of file objects by distributing metadata across multiple servers, reducing the complexity at each individual server while maintaining overall system capacity.
Solution Approach 2:
The system introduces an additional organizational dimension for metadata by using hierarchical or namespace-based routing. File objects are organized not just by storage location but also by metadata server assignment, creating a two-dimensional addressing space. This dimensional organization allows efficient metadata management at scale by providing multiple levels of abstraction and access paths.
Data Source
AI summary
Managing file objects in a data storage system includes providing a plurality of metadata servers, each metadata server having information for only a subset of file objects in the storage system, each subset for each of the metadata servers being less than a total number of file objects for the system and includes providing at least one metadata location server, where an entity accessing a specific file object of the system determines which of the metadata servers contains data for the specific file object by first accessing the at least one metadata location server. Information for the subset of file objects may include a table having file identifiers and metadata location information. The metadata location information may point to data storage for a metadata object for a file.