FlowComb Middleware for Hadoop Network Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The native Hadoop MapReduce scheduler in data-intensive applications fails to effectively manage network congestion, leading to bursty traffic and degraded performance due to limited consideration of network conditions during task assignment.
Innovation Solution
The development of FlowComb, an intelligent network middleware that dynamically reconfigures network flows using OpenFlow to identify and mitigate network congestion by predicting data movement between Hadoop components, thereby optimizing routing and scheduling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the native Hadoop MapReduce scheduler assigns tasks based on server availability, then task scheduling is simple and fast, but network congestion occurs and application performance degrades
Solution Approach 1:
The patent introduces FlowComb, an intelligent network middleware that acts as an intermediary between the Hadoop scheduler and the network infrastructure. FlowComb collects data movement information from the Hadoop JobTracker, analyzes network topology and congestion conditions, and dynamically adjusts flow routing through OpenFlow controllers. This mediator handles the complex network optimization tasks separately, allowing the Hadoop scheduler to remain simple while achieving improved application performance through intelligent network flow management
Solution Approach 2:
The patent segments the scheduling system into two independent components: the Hadoop JobTracker responsible for task assignment based on server availability, and FlowComb responsible for network flow optimization based on real-time network conditions. This segmentation allows each component to specialize in its function without the complexity of the other, resolving the contradiction between scheduling simplicity and performance optimization
2Productivity
If Hadoop MapReduce moves massive data across server racks, then data processing capability is improved, but network congestion and bursty traffic increase
Solution Approach 1:
FlowComb performs preliminary analysis of data movement patterns by collecting information from the Hadoop JobTracker before data transfer occurs. It predicts potential network hotspots and pre-configures optimal routing paths using OpenFlow controllers. This preliminary action prevents network congestion before it occurs, allowing massive data movement to proceed smoothly without causing bursty traffic or performance degradation
Solution Approach 2:
The patent implements dynamic flow routing that adapts to real-time network conditions. FlowComb continuously monitors network topology and congestion status, then dynamically adjusts the routing of individual data flows between Mappers, Reducers, and HDFS nodes. This dynamic adaptation allows the system to maintain high data processing capability while avoiding network congestion by redirecting traffic around congested paths
Data Source
AI summary
A method implemented in a network apparatus used in a network is disclosed. The method comprises collecting information about network topology from a network controller, collecting information about data movement, deciding routing in the network according to the information about network topology and the information about data movement, and providing information about the routing to the network controller, wherein the network controller enforces the routing in the network. Other methods, apparatuses, and systems also are disclosed.