Dataset Image Manager Coordinates Multi-Volume Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cluster storage systems face difficulties in producing a successful overall image of a dataset stored across multiple storage systems, as it is challenging to coordinate and ensure that all systems have successfully produced their images without retaining stale backup images from failed backup attempts.
Innovation Solution
A system and method are introduced that utilize a dataset image manager and volume image managers to coordinate the production of dataset images across multiple storage systems, implementing a start phase to suspend volumes and a commit phase to complete image production, with timeout mechanisms to handle failures and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple storage systems independently produce images without coordination, then each system can complete its image production, but inconsistent states and stale backup images remain across the cluster
Solution Approach 1:
A dataset image manager is introduced as an intermediary component that coordinates image production across multiple storage systems. The manager receives image production requests, distributes them to relevant storage systems, collects status updates, and manages the overall image production lifecycle, thereby ensuring consistency without requiring complex peer-to-peer coordination between storage systems.
Solution Approach 2:
The system suspends I/O operations to dataset volumes before initiating image production and resumes them only after successful completion. This preliminary action ensures that the data being imaged is in a consistent state, preventing stale or inconsistent backup images from being created while maintaining reliability across the cluster.
2Reliability
If I/O operations continue during image production, then data access is maintained, but image consistency cannot be guaranteed
Solution Approach 1:
The system suspends I/O operations to dataset volumes before initiating image production and resumes them only after successful completion. This preliminary action ensures that the data being imaged is in a consistent state, preventing stale or inconsistent backup images from being created while maintaining reliability across the cluster.
Solution Approach 2:
The system dynamically adjusts I/O operation states based on the image production lifecycle. During image production, I/O operations are suspended to ensure consistency; after successful completion, I/O operations are resumed to restore productivity. This dynamic state management resolves the contradiction between reliability and productivity.
3Reliability
If timeout mechanisms are implemented to handle failures, then data integrity is improved, but image production time increases
Solution Approach 1:
The dataset image manager implements timeout mechanisms that provide feedback on image production status. If a storage system fails to complete image production within the timeout period, the manager detects this through lack of status updates and initiates failure handling procedures. This feedback mechanism ensures data integrity by preventing incomplete or inconsistent images from being stored, while the automated failure handling minimizes the time loss compared to manual intervention.
Data Source
AI summary
An application may store data to a dataset comprising a plurality of volumes stored on a plurality of storage systems. The application may request a dataset image of the dataset, the dataset image comprising a volume image of each volume of the dataset. A dataset image manager operates with a plurality of volume image managers in parallel to produce the dataset image, each volume image manager executing on a storage system. The plurality of volume image managers respond by performing requested operations and sending responses to the dataset image manager in parallel. Each volume image manager on a storage system may manage and produce a volume image for each volume of the dataset stored to the storage system. If a volume image for any volume of the dataset fails, or a timeout period expires, a cleanup procedure is performed to delete any successful volume images.