Cloud Controller Snapshot Mechanism for Distributed Filesystem Disaster Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network storage systems face challenges in maintaining data consistency and availability, especially in cloud-based storage solutions, where access latency is high, and network failures can lead to data inconsistency and prolonged downtime.
Innovation Solution
A distributed filesystem architecture where multiple cloud controllers manage and cache data, generating incremental snapshots to ensure data consistency, and a backup controller takes over in case of failures, ensuring seamless data access and high availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If cloud-based storage solutions are used to simplify storage management, then storage capacity and ease of operation are improved, but access latency increases and data consistency problems occur
Solution Approach 1:
The system segments storage management into multiple cloud controllers that independently manage data consistency for different data segments. Each cloud controller maintains its own cache and consistency management, dividing the overall complexity into manageable parts while preserving data consistency through coordinated operations across segments.
Solution Approach 2:
Cloud controllers act as intermediaries between clients and the centralized cloud storage system. They cache data locally and manage consistency within their domains, reducing the need for direct real-time communication with the central storage system and thereby reducing latency while maintaining consistency through their mediation role.
2Quantity of substance
If cloud-based storage solutions are used to increase data storage capacity, then storage capacity is improved, but access latency increases
Solution Approach 1:
Cloud controllers perform preliminary actions by caching frequently accessed data locally before it is needed. They proactively load data into their caches based on access patterns and predictions, so that when clients need data, it is already available locally rather than requiring a latency-prone remote fetch from the central cloud storage system.
Solution Approach 2:
The system segments data access into local cache hits and remote cache misses. By dividing data management across multiple cloud controllers with local caches, the system maximizes the proportion of accesses that can be served locally, thereby improving overall access speed while maintaining access to the large centralized storage capacity through the cloud.
3Adaptability or versatility
If network-based storage systems are used to provide remote storage access, then storage accessibility is improved, but network failures can prevent data access for significant periods
Solution Approach 1:
Cloud controllers perform preliminary action by maintaining local caches of data and generating snapshots of their state. When network failures occur, these pre-loaded caches and snapshots enable clients to continue accessing data locally without requiring continuous network connectivity, thereby maintaining storage accessibility during network outages.
Solution Approach 2:
The system implements beforehand cushioning by having cloud controllers maintain redundant local copies of data and consistency state information. This cushioning buffer absorbs the impact of network failures, allowing the system to operate partially offline while maintaining data availability through the pre-positioned local caches and snapshots.
4Reliability
If multiple cloud controllers manage distributed filesystem data, then data consistency and availability are improved, but system complexity increases
Solution Approach 1:
The system segments consistency management into independent domains handled by individual cloud controllers. Each controller manages consistency for its local cache and associated data without needing to coordinate with every other controller, dividing the complex global consistency problem into simpler local consistency management tasks.
Solution Approach 2:
Cloud controllers maintain local copies of data and metadata snapshots rather than requiring real-time synchronization across all controllers. These copies enable each controller to operate semi-independently, reducing the coordination overhead and complexity of managing consistency across the distributed system while still providing data consistency through the snapshot mechanism.
Data Source
AI summary
The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, a backup cloud controller associated with the distributed filesystem is also configured to receive each (incremental) metadata snapshot, such that, upon determining the failure of a cloud controller, the backup cloud controller can immediately begin receiving data requests from clients associated with the failed cloud controller.