Distributed Memory Cluster Controller for MapReduce Data Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data processing systems using MapReduce face performance degradation due to uneven distribution of intermediate data across storage nodes and inadequate consideration of physical locations and network congestion when allocating reducers.
Innovation Solution
A distributed computing system that employs a controller node to select storage nodes for uniform data distribution and allocate reducers based on physical distance and network congestion, using a node selection rule and memory cluster structure information to manage data across server racks and memory blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the map function randomly selects storage nodes to store intermediate data, then storage node selection is simple, but intermediate data becomes concentrated at one particular storage node degrading system performance
Solution Approach 1:
The patent changes the selection parameter from random selection to selection based on current storage node load status. The controlling node queries the load status of storage nodes and selects nodes with lower load to store intermediate data, thereby balancing the data distribution and preventing concentration at single nodes, which resolves the contradiction between selection simplicity and system performance.
2Device complexity
If the reducer is allocated to map function without considering physical locations or network congestion, then allocation process is simple, but overall performance of the distributed data processing system degrades
Solution Approach 1:
The patent applies preliminary action by having the controlling node query the load status of storage nodes and determine optimal reducer allocation locations before the actual data processing begins. This advance planning considers physical locations and network congestion, allowing the system to optimize data flow paths and avoid bottlenecks, thus resolving the contradiction between allocation simplicity and processing performance.
3Ease of manufacture
If multiple intermediate data portions are concentrated at one storage node, then storage operation is simplified, but bottleneck is created degrading overall system performance
Solution Approach 1:
The patent implements feedback by continuously monitoring the load status of storage nodes and using this information to make dynamic allocation decisions. The controlling node receives load status feedback from storage nodes and adjusts the distribution of intermediate data accordingly, preventing concentration at single nodes and maintaining system performance stability, thus resolving the contradiction between storage simplicity and performance reliability.
Data Source
AI summary
Provided are an apparatus and method for controlling a distributed memory cluster. A distributed computing system may include a computing node cluster, a distributed memory cluster, and a controlling node. The computing node cluster may include a plurality of computing nodes including first computing nodes that each generates associated data. The distributed memory cluster may be configured to store the associated data of the first computing nodes. The controlling node may be configured to select memory blocks of the associated data for distribution on the distributed memory cluster based on a node selection rule and memory cluster structure information, and to select second computing nodes from the computing node cluster based on a location selection rule and the memory cluster structure information.