External Shuffle Service for VM Data Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud computing environments face inefficiencies in data processing across multiple virtual machines, particularly in the shuffle phase of map reduce frameworks, where data partitioning and resource allocation are not optimized, leading to suboptimal performance and inflexibility in managing system resources.
Innovation Solution
Implementing a shuffle service that operates externally to virtual machines, providing an interface for data record partitioning and allocation, allowing data records with the same key to be processed by the same virtual machine, and integrating this service with the file system or exposing it via a network interface, enabling efficient data management and resource sharing across multiple clients.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data partitioning is performed within virtual machines using existing cloud computing environments, then data processing can be distributed across multiple virtual machines, but data partitioning efficiency and resource utilization are suboptimal due to lack of centralized coordination
Solution Approach 1:
The patent introduces a shuffle service as an intermediary component that operates outside the virtual machine environment. This shuffle service provides centralized coordination for data partitioning and allocation across multiple virtual machines, improving data partitioning efficiency without requiring complex coordination logic within each virtual machine. The shuffle service acts as a mediator that manages the distribution of data records to appropriate virtual machines based on partitioning keys.
2Adaptability or versatility
If shuffle service is implemented within virtual machines, then data processing can be performed, but flexibility in implementing and modifying the shuffle service is reduced
Solution Approach 1:
The patent extracts the shuffle service from the virtual machine environment and places it in the host operating system or as a separate external service. This extraction allows the shuffle service to be implemented, modified, and updated independently of the virtual machine software code. The virtual machines interact with the shuffle service through standardized interfaces, enabling flexible implementation changes without altering the underlying virtual machine software.
3Productivity
If data records are not properly partitioned across virtual machines, then data can be processed in parallel, but resource utilization becomes inefficient and performance deteriorates
Solution Approach 1:
The patent implements preliminary partitioning of data records by the shuffle service before distribution to virtual machines. The shuffle service performs data partitioning based on partitioning keys and allocates records to appropriate virtual machines in advance, ensuring optimal resource utilization before parallel processing begins. This preliminary action eliminates the need for virtual machines to perform complex partitioning operations during processing, reducing data partitioning time and improving overall resource utilization.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for clustering for parallel processing. One of the methods includes providing virtual machines with an interface to a shuffle service, the shuffle service executing external of the virtual machines. The method includes receiving data records through the interface, each data record having a key and a value. The method includes partitioning the data records, using the shuffle service, according to the respective keys. The method includes providing a part of the partitioned data records through the interface to the virtual machines, wherein data records having the same key are provided to the same virtual machine. Each of the virtual machines can execute on a host machine and each of the virtual machine is a hardware virtualization of a machine.