HPC Workflow Prediction Models for Provenance Data Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High Performance Computing (HPC) workflows face challenges in efficiently analyzing and optimizing large volumes of diverse provenance data, which is crucial for reliability and reproducibility, but current methods struggle with rapid data ingestion and performance maintenance in distributed environments.
Innovation Solution
The development of methods and apparatus for creating prediction models based on input and output features of HPC workflows, using provenance data to estimate optimal execution plans and optimize workflow execution, including the extraction of relevant features from input and output data and the propagation of output features as input for subsequent activities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If provenance data is continuously stored for every task in HPC workflows to ensure reliability and reproducibility, then data completeness and reliability are improved, but the volume of data grows very quickly and analysis efficiency deteriorates
Solution Approach 1:
The patent extracts only the most relevant features from provenance data using automated feature extraction techniques. Instead of analyzing all raw provenance data, the system identifies and extracts key features that are most indicative of workflow performance and reliability, thereby maintaining reliability while reducing analysis complexity and improving efficiency.
Solution Approach 2:
The patent creates simplified copies of provenance data in the form of feature representations and prediction models. These copies capture the essential information needed for reliability assessment without requiring analysis of the complete raw data, enabling efficient analysis while preserving the reliability information.
2Measurement precision
If all types of diverse data are accessed and analyzed within the workflow to optimize execution, then optimization accuracy is improved, but the complexity of data processing increases
Solution Approach 1:
The patent applies different processing strategies to different types of data based on their relevance and characteristics. The feature extraction process identifies which data sources and features are most important for specific optimization goals, applying targeted analysis rather than uniform processing to all data, thereby improving accuracy while reducing overall complexity.
Solution Approach 2:
The patent transforms diverse provenance data into standardized feature representations with consistent formats and schemas. This parameter transformation enables uniform processing of heterogeneous data types, reducing processing complexity while preserving the information needed for accurate optimization predictions.
3Productivity
If prediction models are created and evaluated for every possible workflow execution plan to find the optimal instantiation, then execution optimization is improved, but the time and computational resources required increase
Solution Approach 1:
The patent pre-computes prediction models during workflow design and planning phases, before actual execution. By creating and evaluating prediction models in advance, the system identifies optimal execution plans beforehand, avoiding the need for time-consuming model evaluation during runtime and thereby reducing the time loss while maintaining execution efficiency.
Solution Approach 2:
The patent evaluates a selective subset of the most promising execution plans rather than exhaustively analyzing every possible plan. The feature extraction and prediction model evaluation focus on key differentiating factors, allowing the system to identify optimal instantiations with partial evaluation, thereby reducing computation time while maintaining optimization quality.
Data Source
AI summary
Methods and apparatus are provided for analytical processing of provenance data for High Performance Computing workflow optimization. Prediction models for a workflow composed of a plurality of activities are created by (i) generating a plurality of prediction functions from input features and output features of the workflow, wherein each of the prediction functions predicts at least one output feature of at least one of activities of the workflow based on the input features of at least one activity; and (ii) combining the plurality of prediction functions to generate the prediction models, wherein each of the prediction models predicts a final output feature of the workflow based on an input of the workflow for a given execution plan of the workflow. A plurality of the prediction models can be evaluated to select, among the possible execution plans, an instantiation of the workflow for a given input that optimizes a given user goal.