Storage System Update Notification for Bandwidth Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transfer methods between distributed sites and central data centers face delays and network bandwidth inefficiencies, as they require complete data copying before analysis can commence, leading to performance degradation and excessive bandwidth consumption.
Innovation Solution
A system where a first storage device at a site sends update notifications to a second storage device, allowing the second device to manage data as the latest version without waiting for complete copying, enabling read requests to access the latest data directly, reducing the need for full data transfer and conserving bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all updating data is transferred to the central data center before analysis, then data completeness is ensured, but analysis start time is delayed and network bandwidth is consumed excessively
Solution Approach 1:
The system performs preliminary actions by transferring only essential metadata (volume names, capacity information, file system types) and update notifications to the central data center before analysis begins. This allows the analysis system to prepare and start processing while remaining data is still being transferred asynchronously, eliminating the need to wait for complete data transfer.
Solution Approach 2:
The data transfer process is segmented into critical metadata transfer (immediate) and full data transfer (asynchronous). The system divides data into essential information needed for analysis planning and the actual data payload, allowing analysis preparation to proceed independently while maintaining data completeness through continuous asynchronous transfer.
2Reliability
If all updating data is transferred to the central data center before analysis, then data completeness is ensured, but network bandwidth consumption increases excessively
Solution Approach 1:
The system extracts and transfers only the essential metadata and update notifications to the central data center, leaving the bulk data transfer to occur asynchronously in the background. This extraction of critical information allows analysis preparation without consuming excessive network bandwidth at any single moment.
Solution Approach 2:
By performing preliminary transfer of metadata and update notifications, the system enables analysis preparation without requiring immediate transfer of all data, thus reducing peak network bandwidth consumption while ensuring data completeness through ongoing asynchronous transfer.
3Ease of operation
If volume unit processing is used for data access, then data access is simplified, but fine granularity management is lost and performance degrades
Solution Approach 1:
The system segments data access into volume-level operations for management simplicity and file-level operations for performance optimization. The management interface operates at volume level for ease of use, while the analysis system can access individual files within volumes, achieving fine granularity performance without sacrificing operational simplicity.
Solution Approach 2:
The system adds a file-level access dimension to the existing volume-level management structure. This allows the same system to provide simplified volume-based management while enabling high-performance file-level access for analysis, effectively operating at multiple granularity levels simultaneously.
4Adaptability or versatility
If data copying is performed between sites for analysis access, then data accessibility is improved, but performance degradation occurs
Solution Approach 1:
The system performs preliminary transfer of metadata and update notifications to enable analysis preparation without immediate data copying. Analysis can begin with local data while remote data transfers occur asynchronously, improving accessibility without the performance penalty of synchronous copying.
Solution Approach 2:
The system uses selective copying of only metadata and update notifications to the central data center, rather than copying entire data sets. This minimal copying approach maintains data accessibility while avoiding the performance degradation associated with large-scale data copying.
Data Source
AI summary
When a second storage device in a second site receives an update notification regarding a first element updated according to a write request from a first host, from a first storage device in a first site, for a second element specified on the basis of the update notification, the second storage device manages data of the first element corresponding to the second element as latest data. The first storage device provides a first volume capable of including a plurality of first elements. The second storage device provides a second volume capable of including a plurality of second elements corresponding to the plurality of first elements. When the second storage device receives a read request from a second host, the second storage device determines whether or not data of the first element corresponding to a read source second element (second element specified from the read request) is the latest data. When a determination result is true, the second storage device acquires the latest data from the first storage device and returns the latest data to the second host.