Kubernetes GPU Scheduling for Partitioned Deep Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing partitioned computing methods face challenges in managing and operating multiple heterogeneous terminals, leading to increased complexity and burden as the number of terminals grows, particularly in efficiently distributing and managing GPU resources.
Innovation Solution
A Kubernetes-based partitioned computing system is proposed, utilizing custom controllers and resources to determine optimal partitioned points and schedule GPU tasks efficiently across multiple terminals and servers, thereby simplifying management and ensuring fair resource distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of heterogeneous terminals increases, then partitioned computing can be applied to more devices, but management complexity and operational burden increase significantly
Solution Approach 1:
The patent introduces a Kubernetes-based server as an intermediary between heterogeneous terminals and the computing workload. The server automatically manages terminal connections, resource allocation, and task scheduling without requiring manual intervention for each terminal. This mediator approach resolves the management complexity by centralizing control and automation.
Solution Approach 2:
The system implements self-service through automatic resource allocation and lifecycle management. The Kubernetes cluster automatically provisions, monitors, and deallocates resources based on terminal needs without manual configuration. The system self-adjusts to handle varying numbers of terminals and their specific requirements autonomously.
2Manufacturing precision
If manual tracking and management of terminal characteristics is performed, then terminal-specific optimization is possible, but time consumption and error probability increase
Solution Approach 1:
The patent replaces manual mechanical management processes with automated Kubernetes-based systems. Instead of manually tracking and configuring each terminal, the system uses programmable resource managers and automated schedulers that dynamically adapt to terminal characteristics. This substitution eliminates time-consuming manual operations while maintaining optimization precision through algorithmic decision-making.
3Ease of operation
If GPU resources are manually allocated to tail models, then resource distribution control is possible, but fairness and efficiency deteriorate as terminal number increases
Solution Approach 1:
The system implements dynamic resource allocation where GPU resources are automatically adjusted based on real-time terminal needs and workload demands. The Kubernetes scheduler dynamically provisions and deprovisions resources as terminals connect and disconnect, ensuring both fairness in resource distribution and efficiency in inference processing. This dynamic approach adapts to changing conditions without manual intervention.
Data Source
AI summary
A Kubernetes-based partitioned computing apparatus considering GPU task scheduling includes a first custom controller that generate a second custom resource corresponding to a plurality of terminals by referencing a first custom resource that defines an optimal partitioned point determination algorithm of a head model executed on a terminal side and a tail model executed on a server side of a deep neural network model to be applied to the plurality of terminals; and a second custom controller that determines a partitioned point of each of the plurality of terminals by referencing the second custom resource and determines GPU scheduling on the server side for a tail model selected according to the determined partitioned point.