Workflow Bottleneck Detection via Provenance Data Disaggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identifying resource bottlenecks in workflow task executions is challenging, especially in concurrent environments where aggregated data makes it difficult to determine which parts of telemetry data refer to specific tasks and aligns telemetry data sequences generated by executions in different infrastructures.
Innovation Solution
The method involves data disaggregation to extract canonical signatures for each task, sequence alignment to map task state sequences under different environments, and sequence analysis to identify bottlenecks by analyzing aligned sequences for sensitivity to changes in resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If aggregated telemetry data is used for multiple concurrent workflows, then data collection is simplified, but it becomes difficult to determine which parts of telemetry data refer to specific tasks and align sequences from different infrastructures
Solution Approach 1:
The patent segments aggregated telemetry data by extracting canonical signatures for individual tasks from mixed workflow data. Each task's execution trace is separated and identified using unique signature patterns, allowing specific task performance to be analyzed independently within the aggregated dataset.
Solution Approach 2:
The patent introduces canonical signatures as intermediary markers that bridge the gap between aggregated telemetry data and specific task identification. These signatures act as unique identifiers embedded in the telemetry stream, enabling differentiation of task-specific data without requiring separate data collection for each task.
2Productivity
If resource allocation is increased to improve workflow execution, then system performance may improve, but it increases cost and resource utilization
Solution Approach 1:
The patent changes the parameter of resource allocation by identifying specific bottlenecks through bottleneck scores and selectively increasing resources only where needed. Instead of uniform resource distribution, the system dynamically adjusts resource parameters based on measured performance bottlenecks, optimizing the balance between productivity and resource consumption.
3Loss of information
If comprehensive telemetry data is collected for all workflows, then more information is available for analysis, but data processing complexity and computational overhead increase
Solution Approach 1:
The patent extracts only the essential information needed for bottleneck detection by generating canonical signatures for each task. Instead of processing all raw telemetry data, the system extracts and retains only the signature patterns that uniquely identify each task's execution characteristics, significantly reducing data processing complexity while maintaining information completeness for bottleneck analysis.
Data Source
AI summary
Techniques are provided for detecting resource bottlenecks in workflow task executions using provenance data. An exemplary method comprises: obtaining a state of multiple workflow executions of multiple concurrent workflows performed with different resource allocation configurations in a shared infrastructure environment; obtaining first and second signature execution traces of a task representing first and second resource allocation configurations, respectively; identifying first and second corresponding sequences of time intervals in the first and second signature execution traces for the task, respectively, based on a similarity metric; and identifying a given time interval as a resource bottleneck of a resource that differs between the first and second resource allocation configurations based on a change in execution time for the given time interval between the first and second signature execution traces. The first signature execution trace may be obtained by disaggregating data related to batches of workflow executions.