Scheduler for Cross-Framework Program Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing parallel computing frameworks face challenges in executing programs written for different frameworks, requiring significant effort to port software between platforms and limiting the combination of different computing models for new use cases.
Innovation Solution
A distributed parallel computing system with a master node and worker nodes, utilizing a scheduler that negotiates with a resource manager to allocate computing resources as containers, allowing foreign programs to execute across multiple frameworks, including MPI and non-MPI compliant systems, through policy-driven resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a program is written for a specific computing framework (e.g., MPI), then it can execute efficiently on that framework, but it cannot execute on frameworks with different protocols and semantics
Solution Approach 1:
The patent introduces a translation layer that acts as an intermediary between foreign programs and the native computing framework. This translation layer converts protocols and semantics from foreign frameworks (e.g., MPI) into the native framework's language, enabling cross-framework execution without requiring programs to be rewritten for each specific framework. The intermediary handles the complexity of framework-specific implementations while presenting a unified interface to users.
2Adaptability or versatility
If computing resources are allocated dynamically to support multiple frameworks, then framework versatility is improved, but resource allocation complexity increases
Solution Approach 1:
The patent implements a universal resource allocation mechanism that can handle multiple computing frameworks through a single unified interface. The resource manager is designed to be framework-agnostic, allocating computing resources based on standardized parameters rather than framework-specific requirements. This multi-functional approach allows the same resource allocation infrastructure to support diverse frameworks without requiring separate allocation mechanisms for each framework.
3Ease of operation
If porting effort is reduced through framework translation, then ease of operation is improved, but translation accuracy and program behavior fidelity may deteriorate
Solution Approach 1:
The patent incorporates feedback mechanisms in the translation layer to ensure accurate conversion between foreign and native frameworks. The system monitors program execution behavior and adjusts translations dynamically to maintain fidelity. Feedback loops verify that translated programs produce expected results, and the translation layer can adapt its conversion strategies based on observed behavior patterns, thereby maintaining program reliability while reducing porting effort.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel computing system to adapt a foreign program to execute on the distributed parallel computing system. The foreign program is a program written for a computing framework that is different from a computing framework of the parallel computing system. The distributed parallel computing system includes a master node computer and one or more worker node computers. A scheduler executing on the master node computer acts as an intermediary between the foreign program and the parallel computing system. The scheduler negotiates with a resource manager of the parallel computing system to acquire computing resources. The scheduler then allocates the computing resources to the worker node computers as containers. The foreign program executes in the containers on the worker node computers in parallel.