Virtual SCSI Layer Optimizes Parallel Data Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel data processing systems, such as Hadoop, face inefficiencies in data deployment due to the need for each data node to transmit and store entire data blocks, leading to excessive storage traffic and redundancy.
Innovation Solution
A virtualization platform with a virtual SCSI layer and interceptor layer inspects storage commands from virtual data nodes, modifies them to include replication indications, and internally replicates data blocks within the storage device, eliminating the need for each node to write data blocks separately, thereby reducing storage traffic and enhancing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the complete data file is deployed to each data node to achieve redundancy and parallelism, then reliability and parallelism are improved, but storage traffic and network bandwidth consumption increase significantly
Solution Approach 1:
The system segments the data deployment process by separating the original data file into multiple data blocks, which are then distributed to different data nodes. This segmentation allows each node to store only the necessary blocks rather than complete redundant copies, reducing overall storage traffic while maintaining parallelism and reliability through distributed block placement.
2Reliability
If each data node transmits and stores entire data blocks separately, then data redundancy is achieved, but storage traffic and deployment time increase
Solution Approach 1:
The system performs preliminary actions by pre-calculating and determining the optimal placement of data blocks across data nodes before actual deployment. The virtualization platform pre-organizes the striping pattern and block distribution strategy, allowing data nodes to directly store blocks without redundant transmission overhead, thus reducing deployment time while maintaining redundancy.
3Productivity
If data is propagated through multiple data nodes in a pipeline fashion, then parallelism is achieved, but network traffic and storage traffic increase excessively
Solution Approach 1:
Instead of propagating data through multiple nodes in a traditional pipeline where each node receives and forwards complete data sets, the system uses copying at the block level. The virtualization platform directly copies data blocks from the source to multiple data nodes simultaneously, eliminating redundant network traffic while maintaining parallel deployment across nodes.
Data Source
AI summary
This document describes techniques for efficient data deployment for a parallel data processing system. In one embodiment, a virtualization platform running a parallel processing application that includes one or more virtual data nodes receives a first command to write a data block to a storage device. The platform then determines whether the first command was sent by a first virtual data node. If the first command was sent by a first virtual data node, the platform then 1) writes, the data block to a first location in the storage device; 2) returns the first location to the first virtual data node and 3) determines whether the data should be replicated. If the data should be replicated, the platform instructs the storage device to make a copy of the data block to a second location in the storage device and storing the second location in a tracking structure.