Job Management with Dynamic Node Allocation for Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems with multiple nodes, memory usage can exceed available capacity, leading to memory shortages and decreased performance due to memory swap operations, especially when executing large-scale jobs.
Innovation Solution
A job management program that dynamically adjusts the number of computing nodes used for job execution by switching between jobs at divided time intervals, maintaining data distribution in memory, and increasing node usage when necessary to prevent memory overflow.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple jobs are executed in parallel using time slice execution, then system productivity is improved, but memory usage increases and may exceed available capacity
Solution Approach 1:
The patent dynamically adjusts the number of computing nodes allocated to each job based on current system memory usage. When memory usage approaches available capacity, the system reduces the number of nodes for subsequent jobs or switches to sequential execution, preventing memory overflow while maintaining optimal parallel execution when memory is sufficient
Solution Approach 2:
The system changes the parameter of node allocation based on memory usage conditions. By monitoring available memory capacity and adjusting the number of computing nodes accordingly, the system optimizes the balance between parallel execution efficiency and memory consumption, resolving the contradiction between productivity and memory usage
2Productivity
If the number of computing nodes is increased for job execution, then job processing capability is improved, but memory shortage occurs
Solution Approach 1:
The system implements feedback control by monitoring memory usage status and adjusting node allocation accordingly. When memory usage exceeds available capacity, the system detects this condition and reduces the number of nodes allocated to jobs, preventing memory shortage and maintaining system reliability
Solution Approach 2:
The patent makes the node allocation dynamic rather than static, adjusting the number of computing nodes based on real-time memory availability. This dynamic adjustment ensures that job processing capability is optimized without causing memory shortage, resolving the contradiction between processing capability and memory availability
3Quantity of substance
If data is stored in distributed manner in memory of each node, then memory utilization is improved, but memory swap operations occur when memory capacity is exceeded
Solution Approach 1:
The system performs preliminary checks of memory capacity before allocating jobs to computing nodes. By predicting whether memory usage will exceed available capacity based on the distributed data storage pattern, the system prevents memory swap operations by adjusting node allocation in advance, maintaining system performance
Data Source
AI summary
A computer-readable recording medium having stored therein a Job management program causes a computer to execute a process includes: obtaining, for each of jobs submitted to a system including computing nodes, a number setting of nodes configured to be used for an execution of the each job; determining whether the number setting of nodes is smaller than or equal to a threshold; increasing, for at least a part of one or more jobs of which number setting of nodes is smaller than or equal to the threshold, a number of the computing nodes used for the execution; and causing at least a part of the computing nodes to execute the jobs by switching between the jobs at divided time intervals while maintaining data of the jobs stored in a distributed manner in a memory in each node of the computing nodes in the system.