Virtual Machine Cloning for Parallel Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional MapReduce frameworks for large-scale data processing are costly and inefficient for users with light to medium data processing needs, as they require large clusters of hardware servers and are prone to machine failures, making them unorthodox and expensive.
Innovation Solution
A system and method for parallel data processing using a dynamically created cluster of virtual machines distributed over a server farm, where a parent virtual machine clones itself to create child VMs, dividing the data set into chunks and processing each chunk to generate intermediate results, which are then reduced using the entire server farm's resources without the need for additional software implementations like Apache Hadoop.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large cluster of hardware servers is used for MapReduce processing, then data processing capability is improved, but cost and complexity increase
Solution Approach 1:
The patent uses virtual machine cloning to create multiple worker nodes from a single template VM. Instead of deploying numerous physical servers, the system creates virtual copies that can be rapidly instantiated and managed, reducing hardware complexity while maintaining processing capability.
Solution Approach 2:
The patent implements a unified VM template that can serve multiple functions - acting as both a worker node for data processing and a potential master node. This multi-functionality reduces the need for specialized hardware configurations and simplifies cluster management.
2Speed
If hardware servers with directly-attached storage are used, then data processing speed is improved, but fault tolerance decreases
Solution Approach 1:
The patent introduces network-attached storage as an intermediary between the VMs and the data. This allows VMs to access data over the network while maintaining fast processing speeds, and simultaneously provides centralized storage that can be replicated and recovered, improving fault tolerance without sacrificing performance.
3Adaptability or versatility
If a dynamic VM cluster is created, then resource utilization is improved, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the master VM automatically manages the creation, configuration, and coordination of worker VMs. The system autonomously handles resource allocation and task distribution, reducing the need for manual intervention and simplifying management despite the dynamic nature of the cluster.
4Productivity
If iterative VM cloning is performed, then parallel processing capability is improved, but overhead time increases
Solution Approach 1:
The patent performs preliminary actions by pre-configuring the VM template with all necessary software, libraries, and configurations before cloning. This allows worker VMs to be rapidly instantiated without requiring post-creation setup, significantly reducing cloning overhead time while maintaining full parallel processing capability.
Data Source
AI summary
Disclosed are systems and methods for parallel processing an input data set. During a map stage of a computation, starting with a first virtual machine (VM) acting as a parent VM and an input data set, the system clones the parent VM to generate at least one linked clone child VM. The system further divides the input data set into a first chunk for the parent VM and a second chunk for the at least one child VM by determining a starting pointer for each chunk. Each chunk is processed by a VM to generate an intermediate data result, which is stored in a network storage device. The plurality of VMs then perform a reduce stage on the plurality of intermediate data results stored in the network storage device.