212 results about "Heterogeneous cluster" patented technology

The invention relates to the technical field of cluster scheduling, and provides a GPU heterogeneous cluster scheduling method and system oriented to large model training and reasoning, which constructs a set of complete cluster scheduling system by integrating multi-source information such as hardware features, running states and historical task data and applying technologies such as a clustering algorithm, a fuzzy comprehensive evaluation method and reinforcement learning. Comprehensive, intelligent and dynamic management and scheduling of GPU cluster resources are realized, the cluster scheduling system can significantly improve the execution efficiency of GPU heterogeneous clusters in large model training and reasoning tasks, the resource utilization rate is improved, the energy consumption is reduced, and the stability and adaptability of the system are enhanced. And an efficient and reliable solution is provided for large-scale deep learning application.

The invention provides a deep learning large model training method and system for heterogeneous equipment, belongs to the technical field of model training, and aims to solve the problem that a heterogeneous GPU cluster cannot be effectively utilized when a traditional scheme is used for training a deep learning large model. Different network layers of a deep learning large model to be trained are divided into a plurality of stages, forward propagation and back propagation calculation of all network layers in each stage is executed by independent virtual equipment, and meanwhile, utilization of GPU resources of different structures is coordinated in combination with a proposed hybrid parallel training strategy, so that efficient model training is realized.

The invention relates to an artificial intelligence accelerator card heterogeneous cluster adaptation method and system and a storage medium, and the method comprises the steps: obtaining a unified standardized adaptation module registered according to a private keyword provided by a training framework; the training framework is a training framework of a heterogeneous cluster model; accessing a hardware layer software stack of each heterogeneous acceleration card through a factory mode in the unified standardized adaptation module; a calculation task distribution mechanism of the training framework is expanded in a plug-in form, so that the training framework autonomously optimizes a target heterogeneous accelerator card; performing condition compiling according to an operation environment where the unified standardized adaptation module is located so as to access a target heterogeneous accelerator card and be used for training framework autonomous optimization; and according to the unified test module, the adaptation state of the unified standardized adaptation module and each target heterogeneous acceleration card is verified, and the problems that the artificial intelligence acceleration card heterogeneous cluster adaptation development cost is high and autonomous optimization cannot be achieved are solved.

The invention relates to a heterogeneous GPU cluster scheduling optimization method and system based on resource affinity, and belongs to the field of computer resource scheduling. Comprising the steps of 1, according to an affinity historical library of historical jobs, matching the historical jobs with relatively high similarity through job characteristics, and estimating resource requirements; 2, for a new job, receiving the job, performing probe testing on each type of GPU, and obtaining and recording job performance data; 3, establishing an affinity evaluation model for each job by using historical information or test data, evaluating affinity values of the jobs under different resource configurations, and providing data support for resource allocation decisions; and 4, entering a scheduling optimization period cycle, constructing an allocation matrix according to a target function of an optimization problem, and allocating corresponding resources to each job. According to the method, accurate resource matching and data support are achieved, the dynamic adaptive capacity is guaranteed, and the system expandability is enhanced.

The invention discloses a large-scale heterogeneous computing power cluster training and pushing acceleration method, which comprises the following steps: S1, collecting operation characteristics of each node of a heterogeneous cluster, and constructing an equipment state parameter set; s2, loading a training and reasoning model, constructing an intermediate representation graph structure, and generating graph structure information; s3, inputting the graph structure information and the equipment state parameter set into a scheduling strategy model constructed based on a neural network ordinary differential equation, and generating an optimization decision result; s4, reconstructing a model graph according to an optimization decision result, and dividing the model graph into a training sub-graph and a reasoning sub-graph; s5, mapping the sub-graphs to heterogeneous nodes, performing resource matching and task distribution according to the equipment state parameter set, and generating a scheduling result; s6, collecting performance feedback information; and S7, inputting feedback information into the scheduling model to update parameters, and adjusting evolution function strategy parameters. According to the method, the scheduling acceleration of the training and pushing task of the deep model in the heterogeneous cluster is realized.

The invention discloses a resource collaborative awareness-based Storm flow calculation dynamic scheduling method, and provides the following technical scheme aiming at the problem that the performance of a Storm default scheduling algorithm is reduced in node abnormity, resource fluctuation and rescheduling scenes: firstly, dynamically sensing node busy, downtime and data flow fluctuation events through a real-time monitoring module; triggering a rescheduling process; secondly, constructing a resource collaborative optimization model based on historical task instance resource requirements, node performance indexes and communication overhead, and generating a task instance allocation scheme by taking minimization of inter-node communication cost as a target and combining CPU / memory dynamic threshold constraints; further, a greedy algorithm is adopted to sort high-relevance task instances, the high-relevance task instances are preferentially distributed to nodes with the optimal historical performance, and it is ensured that the node resource utilization rate does not exceed a dynamic threshold value; meanwhile, node load balancing parameters are corrected in real time through time window smoothing processing, and the remaining resource state is updated; and finally, outputting an optimized topology division result, so that the system delay after rescheduling is remarkably reduced, and the throughput is improved. According to the method, rapid recovery and stable operation of the heterogeneous cluster are realized through resource collaborative modeling, historical data driven dynamic scheduling and load balancing optimization.

The invention discloses a large model training resource configuration optimization method for a cloud heterogeneous GPU cluster. The method comprises heterogeneous GPU resource configuration and resource perception stage deployment. A cost tree pruning mechanism is provided for resource configuration, a plan with low cost is searched preferentially, a configuration scheme is expressed as a tree path, meanwhile, a multi-dimensional pruning method is adopted to reduce unnecessary branches, and efficient GPU configuration is achieved. According to stage deployment, through defining a Compute-Memory Ratio (Compute-Memory Ratio, CMR), the requirements of each stage on the GPU computing power and memory capacity are considered in a combined manner, the communication and computing efficiency in the training process is optimized, and the large model training time is remarkably shortened. And finally, a large model training resource configuration optimization method Espresso oriented to the cloud heterogeneous GPU cluster is provided in combination with resource configuration and stage deployment strategies, and the user cost is reduced to the maximum extent while the training performance is ensured.

The invention discloses a reinforcement learning calculation simulation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence computing, and the method comprises the steps: inputting the determined current model parameter configuration, current hardware configuration and current working load into a target simulation system to obtain a plurality of parallel grouping combinations, determining a target simulation system according to the current hardware configuration, determining an effective parallel packet combination from the plurality of parallel packet combinations based on a preset Monte Carlo method, inputting the effective parallel packet combination into a simulator of a preset neural network model, and performing delay time calculation according to the effective parallel packet combination through the simulator to obtain a delay time sequence; and the combination corresponding to the shortest delay time is used as a target parallel grouping combination, so that the technical problems of mismatching of simulation scenes, insufficient precision and lack of effective support for heterogeneous clusters are solved, reliable performance prediction and optimal parallel strategy suggestions are provided through high-precision performance modeling and automatic exploration, and the method is suitable for large-scale popularization and application. Therefore, the resource consumption of large-scale GRPO training is reduced.

The invention discloses a ground-to-ground heterogeneous cluster formation control method and system based on an adjustable funnel algorithm, and relates to the technical field of distributed formation maneuvering control, and the method comprises the steps: building a dynamic model of a ground-to-ground heterogeneous cluster, and building an undirected graph representing an interaction topological structure of the ground-to-ground heterogeneous cluster; modeling a distributed safety critical formation maneuvering control problem of the air-ground heterogeneous cluster; a fully distributed dynamic compensator facing each follower and a formation maneuvering controller based on an adjustable funnel algorithm are designed, and then a safety-critical control framework based on a disturbance observer is adopted to construct a composite controller; and acquiring a virtual navigator signal and inputting the virtual navigator signal to the constructed various controllers to obtain an actual formation control signal of each follower, and executing distributed safety-critical formation maneuvering control on the followers. Through a distributed safety critical formation maneuvering strategy based on an adjustable funnel method, collision / obstacle avoidance and disturbance suppression of an air-ground heterogeneous cluster can be realized.

The invention relates to a heterogeneous cluster-oriented resource allocation method and device and a storage medium, and the method comprises the steps: executing a plurality of iterations of an initial model on a single node of each heterogeneous cluster, and obtaining the performance characteristics of the model; generating a plurality of parallel strategy combinations comprising the first parallelism degree and the second parallelism degree; generating a plurality of assembly line parallel combinations based on the first degree of parallelism, calculating the load imbalance rate of each assembly line parallel combination, and screening out N combinations with the minimum load imbalance rate from the assembly line parallel combinations; traversing the second parallelism degree and the micro-processing batch in each stage corresponding to each parallel combination to generate a plurality of stage strategy combinations; and calculating video memory occupation and execution time under each stage strategy combination, screening out K stage strategy combinations of which the video memories are smaller than or equal to a video memory threshold value and the execution time is shortest, and carrying out resource allocation on the heterogeneous cluster based on the stage strategy combinations. According to the invention, the problems of low training efficiency and low resource utilization rate are solved.

The invention discloses a deep neural network model parallel reasoning method for a heterogeneous GPU cluster, and relates to the field of distributed machine learning, and the method comprises the steps: obtaining current information, remaining selectable DNN models, deployed DNN models on each GPU server, and GPU servers which do not meet the number constraint of the DNN models; the scheduler selects a DNN model and deploys the DNN model on the selected GPU server, and calculates throughput for executing parallel reasoning at the moment; the combination of the DNN model and the GPU server with the maximum throughput is found, and related information is updated; judging whether the DNN models deployed on the GPU meet the number constraint or not, and updating GPU cluster information until all GPUs meet the specific DNN model number constraint; and repeating the steps until the algorithm converges. According to the method, limited heterogeneous GPU resources are fully utilized, and the DNN model with high compatibility is selected to deploy and execute parallel reasoning, so that the throughput is maximized.

The invention discloses an affine formation maneuvering control method for a heterogeneous cluster system and a related device, and relates to the technical field of cluster cooperative control, and the method comprises the steps: obtaining the state information estimation values of all leaders updated by a target follower at the current moment according to the communication relation of the target follower and a distributed observer, according to the state information of the target follower at the current moment and a preset affine formation maneuvering control protocol, control input of the target follower at the current moment is obtained through calculation; according to the state information of the target follower at the current moment, the control input of the target follower at the current moment and a preset dynamic model, position information of the target follower at the next moment is obtained through calculation; and according to the position information of the target follower at the next moment, an affine formation maneuvering control instruction at the current moment is determined, so that the expected formation configuration of the follower can be achieved when the control instruction is executed.

The invention discloses a hybrid attack defense control method and system for a heterogeneous cluster in an urban confrontation environment, terminal equipment and a medium, and relates to the technical field of unmanned platform cluster control, and the method comprises the steps: constructing an ideal system model of an unmanned platform cluster comprising a leader and a plurality of followers, constructing an information physical hybrid attack model based on the model; using the attack model to simulate an attack behavior of an attacker on an ideal system to obtain an attacked system model; based on an attacked system model, constructing a distributed elastic security estimator with a compensation mechanism, compensating attack influence for each follower and estimating an expected position; and based on the expected position of the follower, constructing a distributed elastic safety controller with a compensation mechanism, and controlling the follower. By designing the estimator and the controller, attack interference is counteracted, control input is generated, and safe collaboration of the heterogeneous nonlinear unmanned cluster under the cyber-physical hybrid attack is guaranteed.

The invention relates to the technical field of formation control of an air-sea heterogeneous cluster system, and discloses a predefined time formation control method based on a fuzzy neural network, and the method comprises the steps: building kinematics and dynamics models of a six-degree-of-freedom four-rotor unmanned plane and a three-degree-of-freedom unmanned ship; establishing a formation tracking error and a speed tracking error of the air-sea heterogeneous cluster system; constructing a preset performance function, and determining an error conversion function; determining a predefined time virtual control law; establishing a predefined time fuzzy neural network, and determining a distributed predefined time formation controller of the air-sea heterogeneous cluster system in combination with a predefined time virtual control law; and performing distributed predefined time error constraint formation control on the air-sea heterogeneous cluster system according to a distributed predefined time formation controller of the air-sea heterogeneous cluster system. According to the method, the robustness of the air-sea heterogeneous cluster is improved, so that the system can converge within the settable predefined time and does not depend on the initial value of the system.

The invention discloses a large model cluster deployment method and device, equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: carrying out the parallel quantification of a large model corresponding to a deployment task based on a preset quantification tool, and carrying out the index evaluation of the quantized model, so as to obtain a corresponding model evaluation result; determining a to-be-deployed model list corresponding to the deployment task based on the model evaluation result; obtaining computing resource information and a cluster running state of a heterogeneous cluster corresponding to the deployment task at present; and determining a target deployment scheme by using the computing resource information, the cluster running state, the to-be-deployed model list and a dynamic planning strategy, and completing a large model deployment operation corresponding to the deployment task based on the target deployment scheme. According to the method and the device, the condition of GPU performance occupation in the heterogeneous cluster can be effectively avoided, so that the deployment cost is reduced, and the operation stability of the large model in the heterogeneous cluster is improved.

An energy-conscious workflow planning system based on LLM with dynamic resource allocation, consisting of: a workflow input interface configured to receive workflow-directed acyclic graphs (DAGs), energy budget constraints, system performance constraints, and natural language requests from human operators; a large language model (LLM) logic module connected to the workflow input interface and configured to analyze the workflow specifications and system constraints in natural language, generate energy-conscious planning recommendations based on the analyzed workflow specifications, and provide explainable planning rationales in natural language; a reinforcement learning-based scheduling unit connected to the LLM reasoning module and configured to: receive scheduling recommendations from the LLM reasoning agent, fine-tune task-resource assignments by dynamically adapting to runtime variations, and perform online resource redistribution under runtime variability; an energy monitoring unit configured to: continuously monitor CPU and GPU utilization in heterogeneous clusters, track power consumption and thermal limits per node, and generate energy profiles for system components; a multi-objective optimization engine configured to: perform a Pareto-optimal scheduling analysis that balances energy consumption, lead time and reliability, apply statistical and AI-supported trade-off analyses and ensure optimal resource allocation based on Pareto frontier analysis; a dynamic resource allocation unit configured to: use predictive models that incorporate LLM inferences and feedback from reinforcement learning, reassign tasks between nodes and clusters while minimizing energy consumption and improving system throughput based on the predictive models; a performance optimization module configured to optimize scheduling decisions using multi-criteria optimization analysis; and a user interface that allows human operators to override and refine planning strategies in real time based on verifiable planning reasons.

The invention discloses an execution time aware computing power network heterogeneous GPU cluster deep learning task node allocation method and system. The method comprises the following steps: firstly, based on a deep learning task, extracting and preprocessing task features and available node features; secondly, a sampler equally divides new tasks without historical data to available nodes, and each node performs mixed sampling on the tasks until all the tasks estimate execution time data; taking execution time data as a training set, taking the task features and the node features as a test set, and using a regression decision tree model to predict the execution time of the task on each node; performing task allocation on each node by using a cost search algorithm and a short job total JCT priority strategy; and finally, periodically monitoring node resources released in the cluster to obtain an optimal node allocation result. According to the method, task delay and total task JCT are remarkably reduced, cluster node resource changes are monitored in real time, and the resource utilization rate is increased.

The invention relates to a model training method and device based on a heterogeneous GPU cluster and a storage medium. The method comprises the steps of obtaining hardware index data of each GPU in the heterogeneous GPU cluster; obtaining a plurality of operation types corresponding to the deep learning algorithm in the to-be-trained model, and measuring operation performance data of operation corresponding to each operation type executed by each GPU; acquiring communication bandwidth multidimensional features and a load state sensing strategy of each GPU; according to the hardware index data, the operation performance data, the communication bandwidth multi-dimensional features and the load state sensing strategy corresponding to each GPU, constructing a multi-dimensional operation performance matrix of each GPU; according to the multi-dimensional operation performance matrix corresponding to each GPU, the GPU is allocated to each structure layer in the to-be-trained model for model training to obtain a target model, the training efficiency of the model is improved, and the training time of the model is shortened.

The invention discloses a heterogeneous cluster task scheduling fusion method based on Q learning and a genetic algorithm, and belongs to the technical field of intelligent scheduling. Collaborative optimization is carried out through a dynamic feedback mechanism of Q-Learning and the global search capability of the genetic algorithm: in an initialization stage, a cluster state space and an action space are defined, and a multi-Q-value table and an initial population are generated; during task scheduling, nodes are selected based on a # imgabs0 #-greedy strategy, reward values are calculated, and a Q value table is updated; meanwhile, the scheduling scheme is coded into chromosomes, a new population is generated through roulette selection, crossover and variation, and a Q value table is optimized; in the iteration process, the convergence is judged according to the Q value change or the population fitness change, and the strategy is dynamically adjusted. According to the method, the advantages of double algorithms are fused, local optimum is avoided, the task processing efficiency and the resource utilization rate are remarkably improved, adaptive strategy updating during cluster state change is supported, and the method is suitable for an efficient task scheduling scene of a large-scale heterogeneous cluster.

The invention discloses a system quick recovery method and system based on system mirror image management, and the method comprises the steps: initializing a boot partition of a functional blade in a heterogeneous cluster system and a boot flag bit of the boot partition, the boot flag bit being used for BIOS reading to select a corresponding boot partition when the functional blade is started; the method comprises the following steps: triggering a job operating system or a boot operating system of a target function blade to restart through a soft shutdown interface, and modifying a start boot flag bit in a BMC (Baseboard Management Controller) of the target function blade, so that the target function blade is sequentially restarted and switched into the boot operating system from the job operating system; and executing system quick recovery or backup for the target system partition under the boot operating system, and then restarting and switching to run under the specified job operating system so as to complete system quick recovery or backup. According to the method, the characteristics of the heterogeneous cluster system are combined, and automatic and flexible system mirror image fast rollback recovery is achieved when the heterogeneous cluster system has a serious fault.

The invention relates to a large model module level training strategy optimization method and device for a heterogeneous cluster and computer equipment, and the method comprises the steps: obtaining the calculation information of a large model module level and the communication information of an operator level for the heterogeneous cluster; the calculation information comprises calculation time information and module video memory information of modules under different data scales under different distributed strategies and on different chips; the communication information is communication time delays of communication operators under different data scales; determining an initial distributed training strategy of each assembly line based on the module video memory information; based on the calculation time information and the communication information, calculation time of different stages of the assembly line is determined; and optimizing each initial distributed training strategy according to the calculation time and a video memory threshold value for bearing the large model equipment to obtain a target distributed training strategy. Through the method and the device, the problem of relatively low resource utilization rate of a large model oriented to heterogeneous clusters is solved.

The embodiment of the invention provides an image processing unit (GPU) cluster communication method and an electronic device, and the method comprises the steps: splitting a GPU cluster, and obtaining at least one algorithm isomorphic group and at least one algorithm heterogeneous group; the protocol distribution communication and the full collection communication are performed through the isomorphic GPUs in each algorithm isomorphic group, and the full protocol communication is performed through the heterogeneous GPUs in each algorithm heterogeneous group. Therefore, through the embodiment of the invention, the problems that the communication traffic between heterogeneous GPUs is increased and extra data copy overhead is introduced due to the difference of communication libraries between different GPUs in a heterogeneous GPU cluster environment can be solved.

Embodiments of the present disclosure relate to a method, an apparatus and a computer program for providing a machine-learning system to be used for classifying transactions as either fraudulent or genuine, and a method, an apparatus and a computer program for classifying transactions as either fraudulent or genuine. The method for generating a machine-learning system for classifying transactions as either fraudulent or genuine based on a plurality of training transactions, each training transaction being associated with labelling information that indicates whether the training transaction is either genuine or fraudulent, comprises clustering the plurality of training transactions into a plurality of clusters based on a similarity measure. The method comprises determining, for each of the plurality of clusters, whether the cluster is homogeneous or heterogeneous. A heterogeneous cluster includes both fraudulent and genuine training transactions. A homogeneous cluster includes either only fraudulent or only genuine training transactions. The method comprises, for each of the heterogeneous clusters, training an associated machine-learning model using the training transactions belonging to the respective heterogeneous cluster.

The invention provides an asymmetric segmentation scheduling system and method in a heterogeneous GPU cluster, and the method comprises the steps: S1, checking the features of an incoming request, and grouping the incoming request into different request buckets according to the token length; s2, in a heterogeneous GPU cluster environment, optimizing a large language model reasoning instance by adopting a double-layer strategy; and S3, calculating the matching degree of the request buckets and the big language model reasoning instances, and scheduling different request buckets to the big language model reasoning instance with the highest matching degree according to a calculation result. According to the method provided by the invention, the model layer can be asymmetrically segmented according to the computing power and the video memory capacity of each GPU on the premise of satisfying the model parallelism degree constraint, the dynamic balance of the execution duration between stages is realized, assembly line cavitation bubbles are fundamentally reduced, and the overall throughput rate is improved.

The invention provides a monitoring method for large model training of a heterogeneous cluster and electronic equipment. The monitoring method for large model training of the heterogeneous cluster comprises the steps of obtaining current training data of a large model on each device in a starting state in the heterogeneous cluster; according to the current training data, determining current performance data used for representing the training process; if the current performance data is not in the normal range, determining fault information according to the historical performance data in the last training process and the current performance data; and determining whether large model training is abnormal or not according to the fault information so as to monitor the training process of the large model. According to the method, the training process of large model training can be monitored, and abnormal conditions such as faults can be found in time.

The invention discloses a distributed constraint formation optimal tracking method, device and equipment and a medium, and relates to the technical field of formation tracking control of clusters, and the method comprises the steps: building a communication topological structure in a heterogeneous cluster system, building a dynamic model of the heterogeneous cluster system, and carrying out the formation tracking control of the heterogeneous cluster system based on the communication topological structure and the dynamic model. Establishing a constraint formation optimal tracking model, designing a distributed constraint formation optimal tracking control protocol, and determining control protocol parameters of the distributed constraint formation optimal tracking control protocol to ensure that a dynamic regret value and a constraint violation value are linearly increased along with time; and based on the distributed constraint formation optimal tracking control protocol and the control protocol parameters, solving the constraint formation optimal tracking model to obtain an optimal formation tracking result, thereby realizing constraint formation optimal tracking.

The invention relates to a heterogeneous cluster cooperative task allocation and path planning method for an agricultural unmanned aerial vehicle, and the method comprises the following steps: S1, carrying out the system modeling and initialization, and carrying out the modeling of a cooperative data collection task of an unmanned aerial vehicle mechanism cluster in an agricultural environment as a decentralized partially observable Markov decision process; s2, defining a multi-dimensional space, wherein the decentralized part observes a global state space, a joint action space and a local observation space of a Markov decision process; s3, designing a mixed reward function; s4, performing cluster strategy training; and S5, performing online distributed execution. The method effectively solves the problems that an existing method is poor in environment dynamic change adaptability, does not fully consider the heterogeneous characteristics of the unmanned aerial vehicles and does not fully consider decision-making and execution separation, and remarkably improves the operation efficiency, safety and collaborative intelligence level of an agricultural unmanned aerial vehicle cluster in a complex unstructured environment.

The invention relates to a heterogeneous cluster performance prediction method, system and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining respective operation data of at least one type of accelerators in a heterogeneous cluster, wherein the operation data is generated when a target model is iteratively trained through the heterogeneous cluster; based on the respective operation data of the at least one type of accelerators, performing statistics on the data processing amount and the data processing duration of the heterogeneous cluster during at least one round of training, and determining the actual measurement force utilization rate of the heterogeneous cluster according to the data processing amount and the data processing duration; an isomorphic cluster matched with the heterogeneous cluster is determined, and the theoretical computing power utilization rate of the isomorphic cluster is predicted according to the total computing power value of the isomorphic cluster and the network structure of the target model; and comparing the actual calculation power utilization rate with the theoretical calculation power utilization rate to obtain a performance prediction result of the heterogeneous cluster. By adopting the method, the heterogeneous cluster performance can be effectively evaluated.

The invention discloses a deep learning model automatic parallel training method suitable for a heterogeneous cluster. The method comprises the following steps: receiving a heterogeneous cluster and a deep learning model required by training as input; for the deep learning model, obtaining execution performance information of the deep learning model on a training system; for a cluster required by training, acquiring execution performance information of the cluster, and generating a maximum isomorphic node set and node set division of the cluster; traversing division of each node set, and solving the maximum throughput of the node set for all the node sets contained in the node set; and then solving the maximum throughput of current node set division through a load balancing algorithm based on an integer quadratic programming method, thereby solving an optimal strategy in all node set division and outputting the optimal strategy. According to the method, the optimal parallel strategy for solving the given deep learning model under the heterogeneous cluster setting is supported. The method can be applied to various distributed training environments such as a single-machine multi-card environment and a multi-machine multi-card environment composed of isomorphic or heterogeneous equipment. Training efficiency is high and energy consumption is low.

Disclosed herein is an apparatus and method for 3D parallelization for a heterogeneous GPU cluster. The method may include generating initialization information based on GPU memory capacity in order to parallelize a model across multiple nodes constituting a heterogeneous GPU cluster, pipeline-parallelizing the model based on the multiple nodes using the generated initialization information, and data / tensor-parallelizing layers of the model, which are allocated to each of the multiple nodes according to pipeline parallelization, based on GPUs mounted in the corresponding node.