Parallel Data Block Transmission for Archive Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data archiving solutions are inefficient in terms of network bandwidth and compute utilization, as they typically download the archive data package, merge data blocks while compute capacity is idle, and then upload the updated package, leading to suboptimal performance and increased storage operations on local hard disks.
Innovation Solution
The solution involves merging and transmitting data blocks in parallel to an archival data store without storing them locally, allowing compute power and network bandwidth to be utilized simultaneously, and transmitting components of the updated archive data package separately to reduce storage on local hard disks and enhance efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data blocks are merged and stored on local hard disk before transmission, then data integrity is ensured, but storage operations on local hard disks increase and processing time increases
Solution Approach 1:
The patent applies preliminary action by sending a static header to the archival data store before transmitting data blocks, enabling the archival store to prepare reception and validation structures in advance. This allows data blocks to be transmitted and validated without requiring complete local assembly first, reducing local storage operations while maintaining data integrity through server-side validation.
Solution Approach 2:
The patent segments the archive data package into a static header and multiple data blocks that can be transmitted separately. This segmentation allows the header to be sent first to establish the data structure, followed by incremental transmission of data blocks that can be validated as they arrive, eliminating the need to store complete data sets locally before transmission.
2Quantity of substance
If data blocks are merged locally before upload, then complete data packages are transmitted, but network bandwidth and compute capacity cannot be utilized simultaneously
Solution Approach 1:
The static header is transmitted in advance to the archival data store, establishing the data structure and enabling the store to begin receiving and processing data blocks immediately. This preliminary action allows parallel utilization of compute capacity for generating subsequent data blocks and network bandwidth for transmitting them simultaneously, improving overall productivity.
Solution Approach 2:
The patent enables continuous useful action by allowing the archival data store to continuously validate and store data blocks as they are transmitted, rather than waiting for complete local assembly. The computing instance can continuously generate and transmit data blocks while the archival store continuously processes them, maximizing both compute and network utilization throughout the entire transfer process.
3Reliability
If the entire archive data package is downloaded and processed locally, then data consistency is maintained, but storage operations on local hard disks increase
Solution Approach 1:
The archive data package is segmented into a static header and incremental data blocks. The static header contains validation information and data structure definitions that enable the archival data store to verify consistency of incoming blocks without requiring complete local assembly. This segmentation shifts validation complexity from local to remote processing, reducing local storage operations while maintaining data consistency.
Solution Approach 2:
The static header acts as an intermediary that carries validation rules and data structure information from the computing instance to the archival data store. This intermediary enables the archival store to perform consistency validation independently, reducing the need for complex local storage operations and validation logic while maintaining data consistency guarantees.
Data Source
AI summary
Technology is described for data archiving in a service provider environment. A first archive data package may be received from an archival data store in the service provider environment. The first archive data package may include first data blocks and first key block data. Second data blocks and second key block data may be received from a source data store in the service provider environment. Merged data blocks may be created by combining the first data blocks with the second data blocks. The merged data blocks may be sent to the archival data store while merged key block data is being created. The merged data blocks and the merged key block data may be included in a second archive data package that is stored at the archival data store.