Delta Information Volumes for Chained Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, particularly in cloud computing environments, existing technologies face challenges in efficiently replicating data across different vendor platforms due to architectural differences and proprietary protocols, often requiring the transfer of entire data volumes even when only minor changes have occurred, which hampers network performance.
Innovation Solution
The method involves generating and storing delta information volumes (DIVs) in a cloud storage system, which identify changed data blocks between snapshots, allowing readers to request and update only the latest changed blocks, reducing the need for complete snapshot transfers and improving data replication efficiency across diverse vendor systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If complete snapshots are transferred between vendor platforms, then data replication accuracy is maintained, but network bandwidth consumption increases significantly
Solution Approach 1:
The patent extracts only the changed data blocks between snapshots using delta information volumes (DIVs) rather than transferring complete snapshots. The DIV identifies specific data blocks that have changed, allowing the reader subsystem to retrieve only those particular blocks from the cloud store, thereby significantly reducing network bandwidth consumption while maintaining replication accuracy.
Solution Approach 2:
The patent segments the data replication process into identifying changed blocks via DIV and transferring only those specific blocks. This segmentation allows the system to avoid transferring unchanged data, reducing overall network traffic while ensuring that all necessary changes are captured for accurate replication.
2Loss of energy
If delta information volumes are used to transfer only changed blocks, then network bandwidth is reduced, but system complexity increases due to DIV generation and processing
Solution Approach 1:
The source subsystem automatically generates DIVs by comparing consecutive snapshots and stores them in the cloud store. This self-service mechanism eliminates the need for manual intervention in tracking changes, and the DIV structure itself serves as the index for identifying changed blocks, simplifying the overall process despite the added data structures.
Solution Approach 2:
The DIV is generated and stored in advance in the cloud store before the reader subsystem needs to replicate data. This preliminary action allows the reader to efficiently query and retrieve only the necessary changed blocks without having to analyze the entire snapshot data, reducing processing complexity at replication time.
3Reliability
If entire data volumes are replicated across vendor platforms, then data consistency is ensured, but replication time increases
Solution Approach 1:
The patent extracts and transfers only the changed data blocks identified by the DIV rather than replicating entire data volumes. The reader subsystem uses the DIV to locate and retrieve only the specific blocks that have changed since the last snapshot, ensuring data consistency while dramatically reducing replication time and network traffic.
Solution Approach 2:
The patent applies partial action by transferring only the necessary changed blocks rather than complete data volumes. This approach performs exactly what is needed (changed block replication) without the excess of transferring unchanged data, optimizing both time and bandwidth efficiency while maintaining consistency.
4Volume of stationary object
If snapshots are stored in cloud store for chained replication, then storage scalability is improved, but access complexity increases for retrieving and updating snapshots
Solution Approach 1:
The DIV is prepared and stored in the cloud store in advance, serving as a pre-computed index of changed blocks. When the reader subsystem needs to replicate data, it can directly query the DIV without having to analyze or compare snapshots, significantly simplifying the access process while maintaining the scalability benefits of cloud storage.
Solution Approach 2:
The DIV acts as an intermediary data structure between the snapshots stored in the cloud and the reader subsystem. It mediates the complex process of identifying changed blocks by providing a structured index that the reader can efficiently query, reducing access complexity while preserving the scalability of cloud-based snapshot storage.
Data Source
AI summary
Method and apparatus for managing data in a distributed data storage system, such as but not limited to a cloud computing environment. In some embodiments, snapshots of a data set are uploaded from a source storage subsystem to a cloud store, along with intervening difference information volumes (DIVs). The DIVs are data structures that identify a set of updated data blocks that have been changed by the source storage subsystem between each successive pair of the snapshots. A reader subsystem requests and uses the latest DIV to request the latest set of changed data blocks from the cloud store, and uses the latest set of changed data blocks to update a previous snapshot to generate a copy of the most recent snapshot. The source and reader subsystems can comprise Internet of Things (IoT) devices, client devices, edge computing devices, etc. from different vendors and which utilize different protocols.