Virtual Machine Data Block Placement for Network Load Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems like HDFS, the process of collecting data blocks from remote physical machines increases network bandwidth load and response time, as data blocks are spread across multiple machines, making efficient data access and processing inefficient.
Innovation Solution
A distribution system that strategically arranges virtual machines and data blocks across physical machines to maximize local access, where data blocks are stored in a single node whenever possible, reducing network load and improving access efficiency by determining the optimal placement of virtual machines based on processing relationships and local access values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data blocks are distributed to multiple physical machines, then data redundancy and fault tolerance are improved, but network bandwidth load increases and response time increases
Solution Approach 1:
The patent applies local quality by making the second virtual machine aware of which data blocks are stored on local physical machines and prioritizing access to those local blocks. This creates different access qualities for different data blocks - local blocks are accessed preferentially while remote blocks are accessed only when necessary, thus reducing overall network bandwidth load while maintaining the redundancy benefits of distributed storage.
Solution Approach 2:
The management machine performs preliminary actions by determining in advance which physical machines should store data blocks based on processing relationships between virtual machines. The system proactively places data blocks on physical machines that are likely to be accessed, so that when the second virtual machine needs data, it can access local blocks without waiting for network transfers.
2Reliability
If data blocks are distributed to multiple physical machines, then data redundancy and fault tolerance are improved, but response time increases
Solution Approach 1:
The system implements local quality by enabling the second virtual machine to identify and preferentially access data blocks stored on local physical machines. This creates a quality differentiation in data access - local blocks provide fast response while remote blocks are accessed only when local copies don't exist, thus maintaining redundancy without uniformly increasing response time.
Solution Approach 2:
The management machine performs preliminary determination of optimal data block placement based on processing relationships. By proactively placing data blocks on physical machines that will be accessed by subsequent processing stages, the system reduces access time while maintaining the fault tolerance benefits of distributed redundancy.
3Productivity
If virtual machines collect data blocks from remote physical machines, then data processing can be performed, but network bandwidth load increases
Solution Approach 1:
The patent applies local quality by enabling virtual machines to differentiate between local and remote data blocks in their access patterns. The second virtual machine is configured to access local data blocks preferentially, which maintains data processing capability while significantly reducing the network bandwidth load that would otherwise be required to fetch all data blocks from remote machines.
Solution Approach 2:
The management machine performs preliminary placement of data blocks on physical machines based on anticipated processing needs. This preliminary action ensures that when data processing occurs, much of the required data is already available locally, reducing the network bandwidth load during the actual processing operation.
Data Source
AI summary
In a distribution system including a plurality of physical machines, when an output file output by first processing performs multistage processing input to second processing, a physical machine which is a deployment destination of a first virtual machine for performing the first processing, a physical machine which is a storage destination of each data block included in the output file, and a physical machine which is a deployment destination of a second virtual machine for performing the second processing are determined based on relationship between the first and second processing.