Pre-started Application Masters for Hadoop YARN Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Hadoop YARN system experiences high latency when clients submit jobs, as they need to dynamically request resources and start application masters (AMs), which is particularly detrimental for small and real-time jobs.
Innovation Solution
A resource management method where the resource manager (RM) pre-selects and encapsulates resources into containers for pre-started AMs, allowing jobs to be submitted to matching AMs for execution without the need for dynamic resource allocation and AM startup, thereby reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If resources are dynamically allocated and AMs are started when a job is submitted, then resource utilization is improved, but job submission latency increases
Solution Approach 1:
The patent pre-starts application masters (AMs) before job submission occurs. The resource manager proactively launches AMs based on predicted job demands and available resources, so that when jobs are submitted, matching AMs are already available to execute them immediately, eliminating the startup latency.
Solution Approach 2:
The system dynamically adjusts the number of pre-started AMs based on current system state, including resource availability, job queue depth, and performance metrics. This allows the system to optimize between resource utilization and latency reduction in real-time conditions.
2Loss of time
If pre-started AMs are used to reduce latency, then job submission latency is reduced, but resource consumption increases
Solution Approach 1:
The patent changes key parameters including the number of pre-started AMs, resource allocation ratios, and scaling thresholds based on system conditions. These parameter adjustments allow the system to optimize resource consumption while maintaining low latency performance.
Solution Approach 2:
The system implements dynamic resource management where pre-started AMs are scaled up or down based on real-time monitoring of job submission rates, resource availability, and system load. This prevents excessive resource consumption while maintaining the latency benefits.
3Productivity
If the number of pre-started AMs is increased, then job execution speed is improved, but system complexity increases
Solution Approach 1:
The patent implements feedback mechanisms where the resource manager monitors job execution performance, resource usage, and system state, then uses this information to automatically adjust the number of pre-started AMs. This closed-loop control simplifies management while maintaining high productivity.
Solution Approach 2:
The system performs preliminary configuration and planning of AM deployment based on historical job patterns and predicted workloads, reducing the need for complex real-time decisions and simplifying overall system management.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Embodiments of the present invention provide a resource management method and apparatus. The resource management method of the present invention includes: selecting, by a resource manager RM, according to a type and a quantity of application masters AMs that need to be started in advance, resources from resources registered by a node manager NM, encapsulating the resources into multiple containers Containers matching the AMs that need to be started in advance, and starting the AMs that need to be started in advance in the matching Containers; and when the RM receives a job submitted by a client, checking, in the AMs that have been started in advance and according to information about the job, whether there is a matching AM to be used to execute the job, and if there is a matching AM to be used to execute the job, submitting the job to the matching AM for execution. In the embodiments of the present invention, it is implemented that, when a client submits a job, the job can be submitted to an AM that has been started in advance for execution, thereby reducing latency resulting from resource allocation and AM startup.