Network-Aware Task Scheduler for Hadoop MapReduce Bandwidth Optimization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Hadoop MapReduce systems face significant execution time delays due to network congestion, as they do not consider network bandwidth requirements when scheduling tasks, leading to inefficient task distribution and prolonged job execution times.

Innovation Solution

A collaborative system is introduced, where a network-aware task scheduler works with a software-defined network (SDN) flow scheduler to directly obtain traffic demands, dynamically updating network information and scheduling tasks based on available bandwidth, thereby reducing execution times by optimizing task distribution across distributed nodes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If Hadoop MapReduce assigns non-local tasks without considering network bandwidth, then task distribution is simple and fast, but execution time increases due to network congestion

Engineering Contradiction:

Improvetask execution speedVSAvoidjob execution time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system implements feedback by continuously monitoring network bandwidth utilization and using this information to dynamically adjust task scheduling decisions. The task scheduler receives real-time network status updates and modifies task assignment strategies accordingly, creating a closed-loop control system that adapts to changing network conditions to optimize execution time

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The task scheduling system transitions from static task assignment to dynamic scheduling based on real-time network conditions. The scheduler continuously adapts its behavior by considering current network bandwidth availability, transforming the scheduling mechanism into a dynamic system that responds to changing environmental conditions rather than following fixed rules

Inventive Principle:

Principle #15Dynamics

2Device complexity

If flow scheduler indirectly estimates traffic demands, then system complexity is reduced, but scheduling accuracy deteriorates leading to poor scheduling decisions

Engineering Contradiction:

Improvescheduling system complexityVSAvoidtraffic demand estimation accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The system introduces an intermediary component that directly communicates with Hadoop MapReduce to obtain accurate traffic demand information. This intermediary acts as a bridge between the flow scheduler and the data processing framework, enabling precise measurement of actual data transfer requirements without significantly increasing overall system complexity

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system replaces indirect estimation mechanisms with direct measurement approaches. Instead of using complex estimation algorithms, the scheduler directly observes and measures actual traffic demands through collaboration with Hadoop MapReduce, substituting mechanical estimation processes with direct sensing and measurement

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS9367366B2System and methods for collaborative query processing for large scale data processing with software defined networking

Publication Date: 2016.06.14 NEC CORP

AI summary

A system includes a task scheduler that works collaboratively with a flow scheduler; a network-aware task scheduler based on software-defined network, the task scheduler scheduling tasks according to available network bandwidth.