Workflow Run-Time Prediction in Distributed Clusters

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for predicting run-times of workflows in distributed computing systems, such as Hadoop or MapReduce frameworks, face challenges due to high variance in job execution times caused by resource contention, leading to inaccurate progress estimates and difficulties in resource allocation and performance optimization.

Innovation Solution

A system and method that utilize past performance characteristics, current executing performance, and cluster contention trends to derive run-time estimates for workflows, providing best case, worst case, and statistical scenarios, and can be integrated with existing distributed compute clusters to offer interfaces for monitoring and resource management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If existing MapReduce frameworks are used in multi-tenant environments, then workloads can be processed in parallel, but execution time prediction becomes inaccurate due to high variance from resource contention

Engineering Contradiction:

Improveparallel processing capabilityVSAvoidexecution time prediction accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The system performs preliminary profiling of workflow performance characteristics and cluster contention patterns before actual execution. Historical data is collected and analyzed in advance to establish baseline metrics and contention models, which are then used to predict execution times without requiring complex real-time calculations during workload execution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system continuously monitors actual workflow execution metrics and compares them against predicted values. This feedback loop allows the system to refine its prediction models by adjusting for observed deviations caused by resource contention, gradually improving prediction accuracy across multiple workflow executions.

Inventive Principle:

Principle #23Feedback

2Adaptability or versatility

If progress estimators are implemented in shared multi-tenant settings, then resource allocation can be optimized, but accurate progress estimates cannot be provided due to unpredictable intra- and inter-tenant resource contention

Engineering Contradiction:

Improveresource allocation optimizationVSAvoidprogress estimate accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The system segments the workflow into discrete tasks and segments resource contention into identifiable contention events. By breaking down the execution timeline into manageable intervals and analyzing contention at each segment level, the system can provide more accurate progress estimates for individual tasks while accounting for overall resource contention patterns.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts prediction parameters based on current cluster state and workload characteristics. By monitoring real-time metrics such as resource availability, contention intensity, and workflow progress, the system modifies its estimation parameters to reflect current conditions, enabling accurate progress prediction despite changing multi-tenant resource dynamics.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If detailed monitoring interfaces are added to track workflow progress, then resource management improves, but system complexity increases

Engineering Contradiction:

Improveworkflow monitoring capabilityVSAvoidsystem integration complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The monitoring system is designed to serve multiple functions simultaneously: it tracks workflow progress, estimates execution times, monitors resource utilization, and detects anomalies. By making the monitoring infrastructure multi-functional, the system avoids adding separate specialized components for each function, thereby reducing overall system complexity while maintaining comprehensive monitoring capabilities.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10831633B2Methods, apparatuses, and systems for workflow run-time prediction in a distributed computing system

Publication Date: 2020.11.10 OPTUM TECH INC

AI summary

A method, apparatus and computer program product predict run-time to completion of workflows executing in a shared multi-tenant distributed compute clusters. The method, apparatus and computer program product receive a MapReduce workflow. The MapReduce workflow includes one or more MapReduce jobs for execution. The method, apparatus and computer program product generates one or more workflow profile summary reports for one or more currently executing workflows on the multi-tenant distributed computing system; generates one or more compact historical workflow profile summary reports of the received MapReduce workflow and the one or more currently executing workflows using the historical reports; generates one or more compact historical cluster contention summary reports of all workflow executing compute clusters in the multi-tenant distributed computing system; and generates a run-time prediction for the MapReduce workflow based at least in part on the one or more currently executing workflow profile summary reports, one or more compact historical workflow profile summary reports, and the one or more compact historical cluster contention summary reports.