22 results about "Data parallelism" patented technology

The embodiment of the invention provides a localized large-scale language model service method and related equipment. The localized large-scale language model service method comprises the following steps: acquiring a request load index of an inference service in real time; according to the request load index, a five-dimensional parallelism strategy is determined in a calculation iteration period, and the five-dimensional parallelism strategy comprises parallelism degrees of five dimensions including data parallelism degree, tensor parallelism degree, pipeline parallelism degree, sequence parallelism degree and expert parallelism degree; and reconstructing the parallel execution mode of the current service into the five-dimensional parallel strategy without interruption, and executing the five-dimensional parallel strategy. According to the technical scheme provided by the embodiment of the invention, the system can be dynamically switched among a plurality of parallel dimensions under the condition of load fluctuation, and the overall calculation cost is minimized on the premise of ensuring the service quality.

The present application belongs to the technical field of cloud computing. Provided are a large-model training system, apparatus and method, and a computing device cluster. In the method, a plurality of computing cards of a large-model training system are grouped according to the fact that checkpoints generated by means of the plurality of computing cards have two states, i.e., being identical and being different, under data parallelism and optimizer parallelism, and checkpoints in shared memories of computing nodes where the plurality of computing cards are located have identical items and different items; and in response to a training resumption instruction, a first computing card comprised in a first computing card group loads identical items from a storage device, and broadcasts same to other computing cards in the first computing card group for training resumption. Since only one computing card is selected from the first computing card group to load a checkpoint from the storage device, the occurrence of a bandwidth bottleneck that is caused by all computing cards in the group simultaneously loading checkpoints from the storage device is avoided, such that the training resumption efficiency can be improved.

The present disclosure relates to a model processing method, device, equipment and computer readable storage medium. The present disclosure obtains a first computation graph corresponding to a to-be-trained model and a parallelization strategy of the to-be-trained model, the parallelization strategy of the to-be-trained model including at least one of pipelining parallelism, model parallelism, data parallelism and operator splitting, adds parallelization information in the first computation graph according to the parallelization strategy of the to-be-trained model, obtains a second computation graph, determines a distributed computation graph according to the second computation graph and a computing resource, and trains the to-be-trained model according to the distributed computation graph. The present disclosure realizes the technology based on computation graph editing to support multiple parallelization strategies, so that multiple parallelization strategies can be integrated in a set of systems, thereby realizing a distributed training framework capable of supporting multiple parallelization strategies.

The invention belongs to the technical field of large model deployment in the power grid field, and provides a large model deployment method and system in the power grid field. The deployment method of the large model in the power grid field comprises the following steps of: based on single parameter memory overhead, single gradient memory overhead, single optimizer state memory overhead, model parameter quantity and data parallelism; according to the relationship between parameters such as the number of working nodes in model parallelism and pipeline parallelism and the memory overhead in the parallel decomposition strategy of calculation and storage collaboration, the parallel decomposition strategy of calculation and storage collaboration with the minimum memory overhead is calculated, and the parallel decomposition strategy of the large model in the power grid field is determined; on the basis of a parallel decomposition strategy of a large model in the power grid field, hardware resource allocation is automatically performed by using hyper-parameter input hardware equipment information and a deep learning framework, then deep learning model structure setting and parallel decomposition scheme configuration are performed by using guidance statements, automatic code translation is completed, and automatic conversion and training of a deep learning model are realized.

The invention provides a system architecture for cross-data center large model training and a resource allocation method and system, and belongs to the technical field of cross-wide area distributed large model training. According to the method, large-scale model cooperative training across multiple data centers can be realized, and the bottleneck that the computing power of a single data center is limited is broken through. Through unified modeling and scheduling of calculation, memory and network resources, task loads can be intelligently allocated according to hardware performance and network bandwidth of different data centers, and efficient collaborative utilization of computing power resources is realized. The training task of the super-large-scale model can be rapidly completed in the heterogeneous computing power environment, and the training time is remarkably shortened. The provided flexible parallelism degree allocation method can be adaptive to different task and resource conditions, the proportion of data parallelism, model parallelism and pipeline parallelism is automatically adjusted, the parallelism efficiency is improved, and the communication overhead is reduced. A training time estimation function is integrated, the overall time delay and resource requirements can be predicted before task execution, and a basis is provided for scheduling decision making.

The application provides an optical circuit switch reconstruction scheduling training acceleration system and method based on a maximum network flow. A heuristic scheme is used to optimize the parallel dimension of distributed training and the grouping of training tasks, reduce the communication flow across an optical circuit switch (OCS), and then, under the optimal grouping, select the optimal OCS connection configuration at the corresponding moment based on a maximum network flow algorithm to accelerate the global training time. By using a heuristic pruning method to optimize the combination relationship of data parallelism, tensor parallelism and pipeline parallelism, and placing the training tasks according to the rack where the GPU is located and the OCS domain, the situation of high-frequency tensor parallel communication across the OCS can be reduced.

The application relates to a GPU-based high-performance multi-party secure computation training method and system, and particularly relates to the field of multi-party secure computation protocols.The application aims to provide a multi-party secure computation training framework with higher parallelism, so as to realize parallelism between different layers of a neural network in a manner of combining data parallelism and model parallelism, and improve the data throughput speed of a training process.The method is a multi-party secure computation training system based on a pipeline flow training method, as shown in Figure 1, the method is designed according to the characteristics that the bottlenecks of linear computation network layers and nonlinear computation network layers in the MPC model training process are calculation and communication respectively, a pipeline flow training method is designed, parallelism between sub-networks is realized, and an optimal sub-network segmentation algorithm is realized to balance the training load between each sub-network.The application provides a multi-party secure computation training framework with higher parallelism, parallelism between different layers of a neural network is realized in a manner of combining data parallelism and model parallelism, and the data throughput speed of a training process is greatly improved.

The application discloses a neural network integrated training method and system based on data parallelism and model parallelism, and the method comprises the following steps: deploying a convolutional neural network structure and a full connection layer segmentation structure of a target neural network on a plurality of training units respectively; completing the calculation of each convolutional neural network structure in a data parallel manner, and completing the calculation of each full connection layer segmentation structure in a model parallel manner; merging the outputs of each full connection layer segmentation structure to obtain a training prediction value; using the training prediction value, calculating the loss function and the back propagation gradient of each full connection layer segmentation structure by an improved Softmax method, and synchronously updating the back propagation gradient to each convolutional neural network structure; repeating the above steps to complete the training of the target neural network. The application can significantly reduce the video memory consumption, reduce the communication overhead, optimize the distribution of the calculation load, provide stronger expansibility, and provide a more efficient and reliable solution for the training of a large-scale neural network while ensuring high efficiency.

This application relates to a method and apparatus for measuring and parallel processing umbilical blood flow ultrasound images. The method includes: accelerating the acquisition of a standard cross-section using data parallelism and convolution acceleration; identifying the region of interest (ROI), X-axis region, and umbilical blood flow spectral envelope of the standard cross-section using digital image processing technology, thereby calculating the peaks and troughs of the umbilical blood flow spectrum and locating a continuous and stable umbilical blood flow spectrum. Further identification of scale points allows for the rapid location of velocity and time scales, enabling efficient calculation of the conversion ratios of the Y-axis velocity scale and the X-axis time scale. Based on this, the calculated continuous and stable umbilical blood flow spectrum, the conversion ratio of the Y-axis velocity scale, and the conversion ratio of the X-axis time scale are used to automatically and accurately measure the correlation coefficient of the umbilical blood flow spectrum. This process requires no manual intervention, significantly improving the calculation efficiency of the umbilical blood flow spectrum correlation coefficient.

The invention provides a heterogeneous computing system and method, electronic equipment and a storage medium. The system comprises a high-performance processor core which issues an instruction stream to an associated processor core through a network-on-chip; the instruction stream comprises a vector instruction and a scalar instruction corresponding to the vector instruction; the associated processor core comprises a control core, a vector control unit and a calculation storage unit; the control core executes the scalar instruction, and when the vector instruction is recognized, the vector instruction is forwarded to the vector control unit; the vector control unit translates the vector instruction into a storage operation primitive; and the computing storage unit executes in-memory computing based on the storage operation primitive, and returns a computing result to the high-performance processor core through the network-on-chip. In the system, the high-performance processor core focuses on the complicated control logic which the high-performance processor core is good at, and the data parallel part is unloaded to the associated processor core, so that the overall calculation efficiency of the workload is effectively improved by distributing different types of workloads to the corresponding processors for processing.

The application provides a protein-ligand affinity evaluation method based on a domestic supercomputing platform, comprising: a domestic supercomputing production environment building step comprising: compiling dependent libraries required for running a deep learning model, and completing framework configuration of SWPyTorch; a deep learning model design and implementation step comprising: constructing a deep learning model based on a protein-ligand affinity evaluation dataset, implementing the deep learning model based on a PyTorch framework under an X86 platform, after model implementation, model transplantation, and transplantation of the deep learning model to the domestic supercomputing platform; a deep learning model parallel optimization step comprising: based on the domestic supercomputing platform, optimizing the deep learning model from data parallelism, calculation parallelism, communication parallelism, operator library optimization, and SWPyTorch multi-node parallelism; and a job submission and running step comprising: configuring computing node resources of the domestic supercomputing platform, activating a dependent environment of the domestic supercomputing platform, and submitting a running job.

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using a topology of combined processor sequences. In one embodiment, the present disclosure includes a computer system comprising a plurality of processor groups, each processor group comprising a plurality of processors. A plurality of network switches are coupled to a subset of the plurality of processor groups. A subset of the processors in a processor group can be configured to form a sequence, and the network switches can be configured to form at least one sequence across one or more of the plurality of processor groups to perform neural network computations. Various configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipeline parallelism, layer parallelism, or a combination thereof.

This invention relates to the field of artificial intelligence technology and discloses a hierarchical optimization method and system for multi-cluster hybrid parallel training strategies. The method includes the following steps: S1: Acquire multi-cluster environment information and AI training task information, and construct a multi-dimensional feature space containing static and dynamic features; S2: Based on the multi-dimensional feature space, determine the applicable execution scale of the task through an AI task training hierarchy partitioning model, wherein the execution scale includes multi-cluster training, single-cluster training, or single-machine training; S3: For tasks determined to be multi-cluster training, generate a set of hybrid parallel training candidate schemes containing data parallelism, tensor parallelism, and pipeline parallelism; S4: Based on a multi-dimensional cost model, select the hybrid parallel scheme with the lowest training cost from the set of candidate schemes for execution; S5: Monitor the training process in real time, and dynamically execute cross-cluster load migration and resource reallocation when an abnormal risk is detected.

An electronic device and a controlling method thereof are provided. The electronic device includes memory, comprising one or more storage media, storing instructions and configured to store information on a neural network model and information on a plurality of resources for performing distributed learning on the neural network model, and a processor communicatively coupled to the memory and configured to perform a parallelism process including pipeline parallelism, data parallelism, and tensor parallelism based on the information on the neural network model and the information on the plurality of resources, wherein the instructions, when executed by the processor, cause the electronic device to acquire a first computation amount when performing the distributed learning from a time when a change in the plurality of resources is detected to a next checkpoint using the plurality of resources before the change, if the change is detected while performing the distributed learning according to a result of performing the parallelism process, perform the parallelism process again based on the information on the plurality of changed resources, acquire a second computation amount when performing the distributed learning from the time when the change is detected to the next checkpoint using the plurality of changed resources, as the result of the parallelism performed again, and perform the distributed learning by a method corresponding to a smaller computation amount of the first computation amount and the second computation amount.

Embodiments of the present disclosure include techniques for processing neural networks. Various forms of parallelism can be achieved using a topology of combined processor sequences. In one embodiment, the present disclosure includes a computer system comprising one or more processor groups, each processor group comprising a plurality of processors. A plurality of network switches are coupled to a subset of the plurality of processor groups. In one embodiment, the switches can be optical network switches. A subset of the processors in a processor group can be configured to form a sequence, and the network switches can be configured to form at least one sequence spanning one or more processor groups of the plurality to perform neural network computations. Various configurations for creating Hamiltonian cycles are disclosed to support data parallelism, pipelined parallelism, layer parallelism, or a combination thereof.

The embodiment of the invention relates to a resilient optimizer state for fully fragmented data parallelization. Systems and methods for fault resilience in distributed training of machine learning (ML) models are provided. Examples include a plurality of compute nodes that store an optimizer slice of the plurality of optimizer slices, and a first compute node of a first optimizer slice that stores an optimizer state. The first compute node may store optimizer slice portions, each of the optimizer slice portions may be received from a respective compute node of the plurality of compute nodes and may be a copy of a portion of a respective optimizer slice of the plurality of optimizer slices stored at the respective compute node. In response to a failure of a compute node of the plurality of compute nodes, the first compute node may update a first optimizer slice with an optimizer slice portion corresponding to the failed compute node, and may train an ML model based on the updated first optimizer slice.

The application provides a mixed parallel-based deep convolutional neural network parallel training method for quickly training a large model with a large data set. The method comprises in-group parallelism and inter-group parallelism. For in-group parallelism, firstly, an automatic model partitioning method is used to divide the model into multiple partitions, and the multiple partitions are placed on different GPUs respectively. Then, the small batch data after partitioning is sent to different GPUs one by one in succession, and the multiple partitions are trained in a pipeline parallel manner. Inter-group parallelism replicates model copies on each group, independently calculates gradients, and then updates the summary of the gradients on the inter-group partition to maximize the overlap of calculation and communication, thereby reducing the gradient synchronization time. Experimental results show that compared with the traditional data parallelism and the most advanced parallel training torchgpipe method, the acceleration performance of the method provided by the application is obviously improved.

The invention provides a multi-task learning method and device, electronic equipment and a storage medium, and relates to the field of model training. In the model training process, in addition to parallel task training, serial training on the tasks is further included, in the model training process, the complexity of data parallel can be simplified through serial training, the problem of node resource waste easily caused in the node parallel process is reduced, and the overall calculation efficiency of model training is improved.

A method for training a model based on data parallelism and a terminal. The model comprises local models trained at training terminals, respectively, and the method comprises: obtaining, by a first terminal, respective training losses of the training terminals; and calculating, by the first terminal, a weighted average of the training losses to obtain a weighted training loss, wherein the weighted training loss is for updating a parameter of the local model trained at each of the training terminals.

The invention provides a large model training system, device and method and a computing device cluster, and belongs to the technical field of cloud computing. In the method, checkpoints generated by a plurality of computing cards under data parallelism and optimizer parallelism have the same state and the different state, the plurality of computing cards of a large model training system are grouped, and the checkpoints in shared memories of computing nodes where the plurality of computing cards are located have the same item and different items. In response to the training recovery instruction, a first computing card included in the first computing card group loads the same item from the storage device and broadcasts the same item to other computing cards in the first computing card group for training recovery. As only one computing card is selected from the first computing card group to load the checkpoint from the storage device, the bandwidth bottleneck caused by the fact that all computing cards in the group load the checkpoint from the storage device at the same time is avoided, the bandwidth of loading the checkpoint from the storage device in the training recovery process can be reduced, the efficiency of loading the checkpoint is high, and the training recovery efficiency is high.

The present application belongs to the technical field of computer application, and particularly relates to a method for accelerating protein-ligand pocket prediction and docking through three-layer optimization. In the method, the data processing layer realizes efficient overlap of calculation and communication through operator space reconstruction and fine-grained task parallelism, and maximizes single-card inference throughput; the model optimization layer performs multi-dimensional analysis through offline feature analysis and dynamic strategy matching, maximizes calculation efficiency, and reduces data handling overhead; the calculation acceleration layer improves the single instruction multiple data parallelism of GPU through tensor layout reconstruction, heterogeneous bias fusion and operator fusion; the operator fusion improves operation efficiency and reduces kernel startup overhead through linear projection fusion, attention calculation fusion and output fusion. The method solves the problems of low overall calculation efficiency of the model and the memory bottleneck of the attention mechanism in the prior art.

The application discloses a large-scale language model automatic parallel training method and system based on a parallel Monte Carlo algorithm, and the method steps comprise the following steps: in step S01, a large-scale language model and a device topology structure of a computing cluster are acquired; in step S02, the acquired large-scale language model is converted into an operator sequence and a corresponding calculation graph, and a model representation of the model is generated; in step S03, a unified parallel strategy search space containing at least three dimensions of data parallelism, model parallelism and pipeline parallelism is constructed according to the operator sequence, the calculation graph, the model representation and the device topology structure; in step S04, an optimal parallel training strategy is searched in the unified parallel strategy search space by using a parallel Monte Carlo tree search algorithm; and in step S05, the large-scale language model is trained according to the optimal parallel training strategy. The application can realize automatic parallel training of the large-scale language model, reduce the time and resources required for training, and improve the training efficiency.