Copy-on-Read Driver for Distributed Data Provisioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face inefficiencies in data transfer from remote storage, leading to increased processing delays and reduced value due to the time-consuming nature of data set transfer, especially in cloud computing environments where hourly rentals are common.
Innovation Solution
Implementing a copy-on-read driver and adaptation layer logic within a distributed computing system's node computer to stream data blocks from remote storage to local storage on demand, allowing for transparent and efficient data provisioning and processing without the need for transferring the entire data set upfront.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data set is transferred to the computing facility before processing, then data availability is ensured, but data transfer time increases significantly, reducing overall processing efficiency and value
Solution Approach 1:
The system performs preliminary actions by setting up the data transfer infrastructure and metadata in advance, but actual data transfer is deferred until needed. The copy-on-read driver is pre-configured to automatically transfer data blocks when processing requests them, rather than transferring all data upfront.
Solution Approach 2:
Instead of transferring the entire original data set, the system creates and transfers only the necessary data blocks on demand. The copy-on-read driver copies individual data blocks from remote storage to local storage as they are needed for processing, rather than copying the complete data set in advance.
2Loss of time
If data is transferred on-demand during processing, then data transfer time is reduced, but system complexity increases due to additional software components
Solution Approach 1:
The copy-on-read driver serves multiple functions: it acts as a traditional file system driver for local storage, simultaneously manages remote storage communication, automatically transfers data blocks on demand, and generates metadata. This multi-functionality reduces the need for separate dedicated components for each task.
Solution Approach 2:
The copy-on-read driver acts as an intermediary layer between the processing system and both local and remote storage systems. It mediates all read requests, intelligently determining whether data should be retrieved from local or remote storage, and automatically managing the transfer process without requiring changes to the processing logic.
3Productivity
If more data is made available for processing, then more information and value can be extracted, but the time required to transfer and prepare the data increases, reducing the time-to-value ratio
Solution Approach 1:
The system enables continuous useful action by allowing processing to begin immediately with available data blocks while simultaneously transferring additional data blocks in the background. The copy-on-read driver ensures that data transfer and processing occur in parallel, maximizing the continuity of productive work without interruption.
Data Source
AI summary
Data comprising a plurality of data blocks is provisioned to a distributed computing system, via a network, from remote data storage at a data-provider site. At a node computer of the distributed computing system, a network access client is provided for obtaining data blocks from the remote data storage via the network, and adaptation layer logic is provided for generating metadata for the data blocks in a file system of the distributed computing system. Also at the node computer, a copy-on-read driver is provided for accessing the remote data storage via the network access client and for accessing local data storage of the node computer. In response to a first read request corresponding to a data block from the node computer, the copy-on-read driver copies that data block from the remote data storage to the local data storage for use in the distributed computing system.