Process Count Determination for HPC Application Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing (HPC) systems face challenges in determining optimal process counts for executing applications efficiently, leading to increased wall clock times and decreased parallelization efficiency due to synchronization overhead and resource allocation inefficiencies.
Innovation Solution
An electronic device determines the wall clock time and parallelization efficiency for various process counts by analyzing function call contexts and training models to select the target process count that minimizes wall clock time while maintaining sufficient parallelization efficiency, using techniques such as latency modeling and reinforcement learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If process count is increased to improve parallelization efficiency, then productivity increases, but wall clock time increases due to synchronization overhead
Solution Approach 1:
The system dynamically adjusts process count parameters based on multiple factors including application characteristics, hardware resources, and workload conditions. By changing the process count parameter adaptively rather than using a fixed value, the system optimizes the balance between parallelization efficiency and synchronization overhead, resolving the contradiction between improved productivity and reduced wall clock time.
2Loss of time
If process count is increased to reduce wall clock time, then speed improves, but parallelization efficiency decreases due to resource allocation inefficiencies
Solution Approach 1:
The system implements dynamic process count adjustment where the number of processes is not fixed but adapts based on real-time conditions including application behavior, resource availability, and performance metrics. This dynamic approach allows the system to optimize wall clock time while maintaining adequate parallelization efficiency by responding to changing conditions rather than relying on static process allocation.
Solution Approach 2:
The system employs feedback mechanisms that monitor parallelization efficiency and wall clock time metrics, then use this information to adjust process count decisions. By continuously measuring performance and feeding this information back into the process selection logic, the system resolves the contradiction between reducing wall clock time and maintaining parallelization efficiency through data-driven optimization.
3Productivity
If process count is manually configured to optimize performance, then productivity improves, but device complexity increases due to explicit input requirements
Solution Approach 1:
The system implements self-service capabilities where the process count is automatically determined based on application characteristics, hardware resources, and workload conditions without requiring explicit user input. The system analyzes job descriptions, profiles application behavior, and autonomously selects optimal process counts, thereby maintaining high execution efficiency while eliminating the complexity of manual configuration.
Solution Approach 2:
The system performs preliminary analysis of application characteristics, hardware capabilities, and workload parameters before execution to pre-determine the optimal process count. By conducting this analysis in advance and caching the results, the system achieves optimized execution efficiency without requiring complex real-time configuration or user input during job submission.
Data Source
AI summary
A process-implemented method includes: based on obtaining a job description on an application, determining a wall clock time of the application according to processes of a corresponding candidate count for each of a plurality of candidate counts; determining parallelization efficiency for each of the candidate counts based on the determined wall clock time and a wall clock time of the application according to a single process; and executing the application with processes of a target count selected based on the determined parallelization efficiency among the plurality of candidate counts.