Distributed Metadata Update Policy for Storage Controllers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, maintaining consistent metadata across multiple controllers while allowing for efficient updates is challenging, especially when dealing with large or simultaneous updates, which can lead to data integrity issues and increased communication burdens.
Innovation Solution
A storage system with distributed metadata management, where updates are handled by sending changes to either individual controllers or a common storage medium based on a policy considering update size, logical locality, and service level agreements, ensuring metadata consistency and reducing communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata updates are sent to all controllers simultaneously, then metadata consistency is maintained, but communication overhead and update latency increase
Solution Approach 1:
The patent segments metadata updates into two categories: small updates that are propagated to all controllers, and large updates that are stored only in the common storage medium. This segmentation allows the system to maintain consistency for critical metadata while avoiding the communication overhead of propagating large updates to all controllers, thus resolving the contradiction between consistency and latency.
Solution Approach 2:
The patent applies local quality by allowing controllers to have different types of metadata copies based on the update size. Controllers maintain full metadata for small updates (ensuring local consistency) while relying on the common storage medium for large updates (reducing local storage requirements and update propagation overhead). This resolves the contradiction by applying different consistency strategies locally based on update characteristics.
2Productivity
If all controllers maintain complete metadata copies, then data access efficiency is improved, but storage resource consumption and update complexity increase
Solution Approach 1:
The patent segments the metadata storage responsibility between controllers and the common storage medium based on update size. Controllers maintain complete metadata for small updates (improving access efficiency) while the common storage medium stores large updates (reducing controller storage burden and update complexity). This segmentation resolves the contradiction between access efficiency and update complexity.
Solution Approach 2:
The patent applies partial action by having controllers maintain only the necessary portion of metadata (small updates) locally, while large updates are handled partially through the common storage medium. This partial maintenance of metadata copies improves access efficiency for frequent small updates while reducing the complexity of propagating and managing all metadata across all controllers.
3Reliability
If large metadata updates are propagated to all controllers, then metadata consistency is ensured, but communication bandwidth consumption and update time increase
Solution Approach 1:
The patent extracts large metadata updates from the controller-to-controller propagation path and stores them directly in the common storage medium. This extraction eliminates the need to consume communication bandwidth for propagating large updates across the controller network, while still ensuring consistency through the common storage medium. This resolves the contradiction between consistency and bandwidth consumption.
Solution Approach 2:
The common storage medium acts as an intermediary for large metadata updates, receiving and storing updates without requiring propagation to all controllers. This intermediary approach ensures that metadata consistency is maintained through the common storage medium while avoiding the communication bandwidth consumption that would result from propagating large updates to all controllers directly.
Data Source
AI summary
A storage system and method of operating the storage system is described where the metadata used to access the data stored in a storage device is distributed amongst a plurality of controllers in communication with a user, with each other and with the storage device. Each controller stores at least the metadata needed to access the data relevant to the user at the time, and metadata is updated to respond to internal system activities such as device failures, snapshots, backup operations or the like. To preserve coherence of the metadata, each metadata update is communicated to the other controllers and the storage device. The update is either transmitted to the other controllers and to the storage device and each metadata location is updated, or the update is transmitted to the storage device and each of the controllers is instructed to request an update from the storage device.