Dynamic Resource Sharing Between Mesos and YARN via Bridge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Datacenters face inefficiencies due to statically partitioned resources when running multiple resource allocation systems independently, such as Mesos and YARN, which limits resource sharing and optimal allocation.
Innovation Solution
A non-intrusive open architecture that combines Mesos and YARN resource managers to enable resource sharing without modifying existing protocols, leveraging cgroups for hierarchical task management and using a Control Plane to make informed decisions on resource allocation, allowing both systems to coexist and share resources dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple resource allocation systems (Mesos and YARN) run independently, then each system can manage its own resources, but resources become statically partitioned and cannot be shared between systems
Solution Approach 1:
The patent merges Mesos and YARN resource allocation systems into a unified architecture where both systems can coexist and share the same physical infrastructure. A bridge component enables resource negotiation and allocation between the two systems, allowing dynamic resource sharing while maintaining the operational independence of each system.
Solution Approach 2:
The patent introduces a bridge component as an intermediary between Mesos and YARN systems. This mediator facilitates resource allocation decisions, enables cross-system resource sharing, and coordinates task scheduling without requiring modifications to the core protocols of either Mesos or YARN.
2Productivity
If resources are statically partitioned between multiple resource managers, then each resource manager has dedicated resources, but overall cluster utilization decreases
Solution Approach 1:
The patent implements dynamic resource allocation where resources are not fixed to specific resource managers but can be dynamically assigned based on demand. The bridge monitors resource usage and task requirements in real-time, enabling resources to be allocated to whichever system needs them most at any given moment, thereby maximizing cluster utilization and preventing resource idle time.
3Adaptability or versatility
If existing resource allocation protocols are modified to enable sharing, then resource sharing becomes possible, but system stability and compatibility are compromised
Solution Approach 1:
The bridge acts as an intermediary layer that translates between Mesos and YARN protocols without modifying either system's core protocol implementation. This maintains protocol compatibility and system reliability while enabling resource sharing through standardized interface definitions and translation mechanisms.
Solution Approach 2:
The patent segments the resource allocation functionality into distinct layers: protocol-specific components remain unchanged for compatibility, while the bridge implements sharing logic at an intermediate layer. This segmentation allows resource sharing capabilities to be added without affecting the stability or compatibility of existing protocol implementations.
Data Source
AI summary
In various example embodiments, a system and method for managing a server cluster are provided. An example method may include scheduling a first job on a first node, using a first resource manager, establishing a service for a second resource manager on a second node, wherein the service is allocated node resources of the second node, and attempting to schedule a second job on the first node, using the first resource manager. The method may include preempting the service on the second node, using the second resource manager, in response to the attempt to schedule the second job on the first node, and deallocating the node resources of the second node from the service. The method may include advertising, using a node manager of the first resource manager, available node resources of the second node, and scheduling the second job on the second node, using the first resource manager.