Memory-Tracking Resource Manager for Distributed Graph Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph processing systems face challenges in efficiently managing memory across multiple machines, particularly when handling large graphs and graph operations that produce unpredictable amounts of output data, leading to issues like live-locks and sub-optimal resource allocation.
Innovation Solution
A resource manager design that employs memory reservations, including pre-reservations, flexible reservations, and fixed reservations, to track and allocate memory centrally, ensuring that graph operations can express their memory requirements accurately and avoid live-locks, while allowing for elasticity in system scaling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If memory is allocated dynamically without reservations in distributed graph processing, then system flexibility and elasticity are improved, but memory management stability and job completion reliability deteriorate due to unpredictable memory availability
Solution Approach 1:
The patent implements pre-reservation of memory resources before graph processing jobs are submitted. The resource manager reserves memory in advance based on job requirements and system capacity, ensuring that memory is available when needed. This preliminary action prevents memory allocation failures during job execution while maintaining system elasticity through controlled reservation release.
2Productivity
If memory is over-allocated to maximize parallelism, then system performance and productivity are improved, but system stability deteriorates due to memory exhaustion and live-locks
Solution Approach 1:
The patent implements dynamic memory reservation adjustment based on real-time system state. The resource manager monitors memory usage and job progress, dynamically increasing or decreasing reservations to optimize parallelism while preventing memory exhaustion. This dynamic control allows the system to maximize productivity without compromising stability.
Solution Approach 2:
The resource manager implements feedback mechanisms that monitor memory consumption and job execution status. Based on this feedback, the system adjusts memory reservations in real-time, releasing memory from completed jobs and allocating it to pending jobs. This feedback loop prevents live-locks by ensuring memory is continuously available for progressing jobs.
3Reliability
If memory reservations are made early and fixed, then job execution reliability is improved, but system adaptability and resource utilization efficiency worsen
Solution Approach 1:
The patent transforms fixed memory reservations into dynamic, adjustable reservations. The resource manager allows memory reservations to be modified based on system state and job progress, converting static allocations into flexible resources that can be reallocated as needed. This maintains execution reliability while improving resource utilization efficiency.
Data Source
AI summary
A resource manager tracks the amount of available memory for a cluster of machines and for each machine in the cluster. The resource manager receives a reservation request from a job for a graph processing operation. The reservation request specifies an identification of the job, a type of reservation, and an amount of memory requested. The resource manager determines whether to grant the reservation request based on the type of reservation, the amount of memory requested, and the amount of available memory in the cluster or in one or more machines in the cluster. In response to determining to grant the reservation request, the resource manager sends a response to the job indicating an amount of memory reserved and adjusts the amount of available cluster memory and the amount of available machine memory for at least one machine in the cluster based on the amount of memory reserved.