Cloud Data Replication via Segmented File Locking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud platforms face challenges in maintaining high availability and low latency data replication across multiple availability zones, particularly during network connectivity issues or hardware failures, which can lead to data synchronization disruptions and service availability reductions.
Innovation Solution
Implementing a system that manages data replication in a cloud environment with multiple availability zones, using a persistence interface to lock files, iteratively store and replicate file portions synchronously or asynchronously based on connectivity status, and utilizing an in-memory data grid for metadata management to ensure synchronized content across zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous replication is used to ensure data consistency across availability zones, then data reliability is improved, but network bandwidth consumption increases and latency is worsened
Solution Approach 1:
The patent segments large files into smaller chunks or blocks for replication. Instead of replicating entire files at once, the system divides files into manageable segments that can be transferred incrementally, reducing the impact on network bandwidth and latency while maintaining data consistency across availability zones.
Solution Approach 2:
The system dynamically adjusts replication timing and mode based on network conditions. When network connectivity is good, synchronous replication ensures data consistency. When network conditions deteriorate, the system can switch to asynchronous replication or buffer segments locally, adapting to real-time network status to balance reliability and latency.
2Reliability
If data replication is performed across multiple availability zones to improve availability, then system reliability is improved, but complexity of data synchronization increases
Solution Approach 1:
The patent introduces an intermediary component (such as a coordination service or metadata manager) that manages replication state and coordinates synchronization across availability zones. This intermediary handles the complexity of tracking which segments have been replicated, resolving conflicts, and managing failover, thereby simplifying the overall system architecture while maintaining high availability.
Solution Approach 2:
The system creates copies of data segments across multiple availability zones and uses checksums or hashing to verify integrity. Instead of complex real-time synchronization protocols, the system relies on simple copy-verify mechanisms where each zone maintains independent copies that can be validated through cryptographic hashes, reducing synchronization complexity.
3Reliability
If file locking is implemented to prevent data corruption during replication, then data integrity is improved, but access speed deteriorates
Solution Approach 1:
The patent applies segmentation to file locking as well, locking only the specific segments being replicated or modified rather than the entire file. This fine-grained locking approach allows other processes to access different segments of the same file simultaneously, maintaining data integrity for the locked segments while preserving overall file access speed through parallel operations.
Data Source
AI summary
The present disclosure relates to computer-implemented methods, software, and systems for managing data replication. A request associated with storing content of a file is received at a storage service provided by in a multiple availability zone cloud platform. A lock request is sent to an in-memory data grid at a first instance of the storage service to lock the file for accessing. An input stream of the file is received at the persistence interface to be read iteratively in portions. A read portion of the file is iteratively stored in a first file system storage associated with instances of the storage service at a first availability zone. The portions of the file are provided iteratively to a replication executor at the first instance of the storage service to request replication of the content of the file into a second file storage of a second availability zone of the cloud platform.