Model acquisition method, apparatus, and chip

Abstract

The present application belongs to the technical field of artificial intelligence. Disclosed are a model acquisition method, an apparatus and a chip. The method comprises: acquiring a plurality of pieces of first data, the plurality of pieces of first data being data generated during a first training process of a large model; on the basis of data features of the plurality of pieces of first data, acquiring a plurality of first groups, each first group comprising some of the plurality of pieces of first data; respectively compressing the first data comprised in the plurality of first groups to obtain a plurality of pieces of second data; and, on the basis of the plurality of pieces of second data, executing a second training process of the large model. In the present application, adaptive grouping can be performed on the basis of the data features of the plurality of pieces of the first data, the plurality of first groups obtained by means of grouping are relatively accurate, and the first data comprised in the plurality of first groups are respectively compressed, thereby reducing compression errors, reducing memory occupation, and ensuring the training efficiency and training effect of the large model.

Description

Model acquisition methods, devices and chips

[0001] This application claims priority to Chinese Patent Application No. 202510469217.5, filed on April 14, 2025, entitled “Model Acquisition Method, Apparatus and Chip”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to model acquisition methods, devices and chips. Background Technology

[0003] With the continuous development of artificial intelligence (AI) technology, the types of AI models are also gradually increasing, with large models being one of them. A large model is a deep learning model that includes a large number of model parameters, meaning that the number of model parameters reaches billions, tens of billions, hundreds of billions, or even more. How to obtain large models has become a problem worthy of attention. Summary of the Invention

[0004] This application provides a model acquisition method, apparatus, and chip to achieve the acquisition of large models. The technical solution provided by this application includes the following aspects.

[0005] Firstly, a model acquisition method is provided, in which multiple first data points generated during the first training process of a large model are acquired. Multiple first groups are obtained based on the data characteristics of the multiple first data points, each first group including a portion of the multiple first data points. The first data points included in the multiple first groups are compressed to obtain multiple second data points. A second training process of the large model is then performed based on the multiple second data points.

[0006] Among them, the data features are the features inherent to multiple first data. Based on the data features of multiple first data, adaptive and accurate grouping can be achieved. Compression processing is performed on the first data included in multiple first groups to reduce compression error and reduce the memory occupied by multiple first data. This increases the memory available for the model parameters and hyperparameters of the large model, which is beneficial to improving the training effect and training efficiency of the large model.

[0007] In one possible implementation, the difference between the first data included in each first group is less than a first threshold.

[0008] The values ​​of the first data in each first group are relatively close, resulting in a small compression error, which ensures the training accuracy of the large model and thus guarantees the training effect of the large model.

[0009] In one possible implementation, the number of bits of the second data of the first type is greater than the number of bits of the second data of the second type, and the intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group.

[0010] In other words, the larger the intra-group difference in the first group, the larger the number of bits in the second data obtained by compressing the first data included in the first group. Since a larger intra-group difference in the first group may result in a larger compression error, a smaller compression force is used to achieve a larger number of bits in the compressed second data, thus minimizing compression error and ensuring the training accuracy and effectiveness of the large model.

[0011] In one possible implementation, the first training process includes a forward computation process, the second training process includes a backward computation process, multiple first data points include activations, and multiple second data points are used to obtain the corresponding gradients during the backward computation process. The number of bits for the third type of second data is greater than or equal to the number of bits for the fourth type of second data, and the gradient corresponding to the third type of second data is backpropagated during the backward computation process. Optionally, the gradient corresponding to the fourth type of second data is not backpropagated during the backward computation process.

[0012] When the gradient corresponding to the activation propagates back through the reverse computation process, the activation is highly sensitive to compression; that is, compression may introduce significant errors because these errors propagate back through the gradient during the reverse computation. Therefore, a smaller compression force can be used to achieve a larger number of bits in the resulting second data.

[0013] In one possible implementation, the first training process includes a forward computation process, which includes multiple ordered forward computations, and multiple first data including activations. The number of bits of the second data of the fifth type is greater than or equal to the number of bits of the second data of the sixth type, and the order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type, the order being the order of the forward computations used to generate the first data in the forward computation process.

[0014] The higher the order of a forward computation, the later the forward computation is in the forward computation process. The later the forward computation is, the higher the sensitivity of the activation generated by the forward computation is to compression. That is, the error that may be caused by compression is larger. Therefore, a smaller compression force is used to make the number of bits of the second data obtained by compression larger.

[0015] In one possible implementation, the method further includes: receiving multiple third data, which are data obtained by compression; decompressing the multiple second data and the multiple third data; and summing the decompressed data.

[0016] This implementation method decompresses the second and third data before summing, resulting in high-precision data. Therefore, the summation is performed on high-precision data, avoiding the numerical overflow problem caused by summing low-precision data. This ensures summation accuracy, reduces summation error, and thus guarantees the training accuracy and effectiveness of the large model.

[0017] In one possible implementation, the method further includes: obtaining at least one second group based on data features of multiple first data, each second group including a portion of the first data from the multiple first data, the difference between the first data included in the second group being greater than a second threshold; and performing a second training process of a large model based on the multiple second data, including: performing a second training process based on the multiple second data and the first data included in the second group.

[0018] In this approach, the larger the intra-group difference in the second group, the greater the potential error from compression. If the intra-group difference in the second group exceeds a second threshold, the first data included in the second group will no longer be compressed, thus avoiding compression errors and ensuring the training accuracy and effectiveness of the large model.

[0019] In one possible implementation, the multiple first data include at least one of activation, gradient, or momentum.

[0020] Activation, gradient, and momentum are data generated during the training of large models. This application compresses at least one of these data, which can reduce the memory occupied by these data and improve the training efficiency and training effect of large models.

[0021] Secondly, a model acquisition device is provided, which includes an acquisition module, a processing module, and an execution module.

[0022] The acquisition module is used to acquire multiple first data points, which are data generated during the first training process of the large model.

[0023] The acquisition module is also used to acquire multiple first groups based on the data characteristics of multiple first data, and each first group includes a portion of the first data from the multiple first data;

[0024] The processing module is used to compress the first data included in the first group respectively to obtain the second data;

[0025] The execution module is used to perform a second training process for the large model based on multiple second data sets.

[0026] In one possible implementation, the difference between the first data included in each first group is less than a first threshold.

[0027] In one possible implementation, the number of bits of the second data of the first type is greater than the number of bits of the second data of the second type, and the intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group.

[0028] In one possible implementation, the first training process includes a forward computation process, the second training process includes a backward computation process, multiple first data points include activations, and multiple second data points are used to obtain the corresponding gradients during the backward computation process; the number of bits of the third type of second data is greater than or equal to the number of bits of the fourth type of second data, and the gradient corresponding to the third type of second data is backpropagated during the backward computation process. Optionally, the gradient corresponding to the fourth type of second data is not backpropagated during the backward computation process.

[0029] In one possible implementation, the first training process includes a forward computation process, which includes multiple ordered forward computations, and multiple first data including activations; the number of bits of the second data of the fifth type is greater than or equal to the number of bits of the second data of the sixth type, and the order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type, and the order is the order of the forward computations used to generate the first data in the forward computation process.

[0030] In one possible implementation, the acquisition module is further configured to receive multiple third data, which are data obtained by compression; the processing module is further configured to decompress the multiple second data and the multiple third data, and sum the decompressed data.

[0031] In one possible implementation, the acquisition module is further configured to acquire at least one second group based on the data features of multiple first data, each second group including a portion of the first data from the multiple first data, and the difference between the first data included in the second group is greater than a second threshold; the execution module is further configured to perform a second training process on the multiple second data and the first data included in the second group.

[0032] In one possible implementation, the multiple first data include at least one of activation, gradient, or momentum.

[0033] Thirdly, a chip is provided, comprising a processor and an interface circuit, the interface circuit being used to provide program instructions to the processor, the processor being used to execute the program instructions, causing a computer with the chip installed to perform the model acquisition method provided in the first aspect or any possible implementation thereof.

[0034] Fourthly, another chip is provided, which includes an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the computer with the chip installed executes the model acquisition method provided by the first aspect or any possible implementation of the first aspect.

[0035] Fifthly, a computer device is provided, the computer device including a memory and a processor, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to cause the computer device to perform the model acquisition method provided in the first aspect or any possible implementation of the first aspect.

[0036] Optionally, there may be one or more processors and one or more memories.

[0037] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0038] In a sixth aspect, a computer-readable storage medium is provided that stores a program or instructions, which, when executed on a computer, cause the computer to perform the model acquisition method provided in the first aspect or any possible implementation thereof.

[0039] In a seventh aspect, a computer program or computer program product is provided, the computer program or computer program product including computer program code, wherein when the computer program code is run by a computer, the computer executes the model acquisition method provided in the first aspect or any possible implementation thereof.

[0040] The model acquisition device provided in the second aspect can be the chip provided in the third or fourth aspect, or the computer device provided in the fifth aspect. The technical effects achieved by the technical solutions and corresponding possible implementations of the second to seventh aspects of this application can be found in the technical effects achieved by the first aspect and its corresponding possible implementations described above, and will not be repeated here. Attached Figure Description

[0041] Figure 1 is a schematic diagram of the training process of a large model provided in an embodiment of this application;

[0042] Figure 2 is a schematic diagram of a chip cluster provided in an embodiment of this application;

[0043] Figure 3 is a schematic diagram of a cumulative activation provided in an embodiment of this application;

[0044] Figure 4 is a schematic diagram of a non-cumulative activation provided in an embodiment of this application;

[0045] Figure 5 is a flowchart of a compression activation method provided in an embodiment of this application;

[0046] Figure 6 is a schematic diagram of a grouping method provided in an embodiment of this application;

[0047] Figure 7 is a schematic diagram of the probability density functions of a normal distribution and a skewed distribution provided in an embodiment of this application;

[0048] Figure 8 is a schematic diagram of another grouping provided in an embodiment of this application;

[0049] Figure 9 is a schematic diagram of an addition calculation based on high-precision format data provided in an embodiment of this application;

[0050] Figure 10 is a schematic diagram of an addition calculation based on low-precision format data provided in an embodiment of this application;

[0051] Figure 11 is a schematic diagram of a reduce scatter provided in an embodiment of this application;

[0052] Figure 12 is a schematic diagram of a calculation method equivalent to reduce scatter provided in an embodiment of this application;

[0053] Figure 13 is a schematic diagram of an all reduce method provided in an embodiment of this application;

[0054] Figure 14 is a schematic diagram of a calculation method equivalent to all reduce provided in an embodiment of this application;

[0055] Figure 15 is a schematic diagram of a model parameter update process provided in an embodiment of this application;

[0056] Figure 16 is a comparative schematic diagram of a training loss provided in an embodiment of this application;

[0057] Figure 17 is a comparative schematic diagram of another training loss provided in the embodiments of this application;

[0058] Figure 18 is a flowchart of a model acquisition method provided in an embodiment of this application;

[0059] Figure 19 is a schematic diagram of a model acquisition device provided in an embodiment of this application;

[0060] Figure 20 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0061] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0062] A large model is a deep learning model that includes a large number of model parameters, meaning the number of model parameters reaches billions, tens of billions, hundreds of billions, or even more. The training process of a large model is used to determine the values ​​of each model parameter, thereby obtaining the large model. The training process of a large model includes multiple rounds of training, each round of training including a forward computation process, a backward computation process, and a model parameter update process, as illustrated in Figure 1.

[0063] Before the first round of training begins, the model is initialized, and the values ​​of each model parameter are set to initial values. These initial values ​​can be configured based on experience or actual needs. Additionally, a training set is acquired, which includes training data, and the training function is the labeled data (actual values).

[0064] The training data is input into the large model to begin the first round of training. During the forward computation of this first round, the model parameters, initially set to their initial values, are computed to obtain activations (predicted values). These activations are intermediate results used in the backward computation. During the backward computation of this first round, the activations and the loss function of the large model are computed to obtain the gradient. The loss function indicates the difference between the predicted and actual values; a smaller loss function value indicates better predictive performance of the large model. The gradient is the partial derivative of the loss function with respect to the model parameters, indicating the direction of parameter updates. During the parameter update process of this first round, the optimizer updates the initial model parameters based on the gradient, resulting in initial values ​​for each parameter. Optionally, the optimizer determines momentum based on the gradient, and the initial model parameters are updated based on this momentum.

[0065] After the first round of training, the second round of training begins. Training data is input into the large model to start the second round of training. During the forward computation of the second round of training, the activations are calculated based on the training data and the model parameters with the first value. During the backward computation of the second round of training, the gradients are calculated based on the activations and the loss function to obtain the gradients. During the model parameter update of the second round of training, the optimizer updates the model parameters with the first value based on the gradients to obtain the model parameters with the second value. Optionally, the optimizer determines the momentum based on the gradients and the momentum obtained in the first round of training. That is, the momentum is updated based on the gradients to obtain the momentum, and the optimizer updates the model parameters with the first value based on the momentum. By updating the momentum, the gradients obtained in each round of training can be accumulated, reducing oscillations during model parameter updates. Momentum includes, but is not limited to, first-order momentum and second-order momentum.

[0066] This process continues for multiple rounds until a certain round of training is completed. If the value of the loss function meets the requirements (e.g., it is less than the first reference threshold), then the large model is considered to have the expected predictive performance and has converged, thus achieving the acquisition of the large model.

[0067] Based on the above description of the large model training process, the terminology is introduced as follows.

[0068] Computing cards are chips used to perform calculations on AI models (such as large models), such as neural processing units (NPUs) or graphics processing units (GPUs).

[0069] Distributed large model training uses multiple computers (including one or more computing cards) or multiple computing cards working together to perform large model computations in order to achieve the training process of large models.

[0070] Model parallelism is a technique applied to the training of large distributed models. When the number of model parameters in a large model is too large, a single computing card cannot handle it. Therefore, the large model is split into multiple computing cards to handle the training.

[0071] Information tensors are tensors that are transmitted between different computing cards during the training of distributed large models, including but not limited to activations, gradients, and model parameters mentioned above.

[0072] Collective communication: During the training of a distributed large model, different computing cards jointly perform computations to train the model. Therefore, parallel computation and communication between these cards are necessary. For example, different cards need to communicate and transmit training data, intermediate results of parallel computations, and information tensors. Exemplary methods of collective communication include, but are not limited to, all reduce, reduce scatter, all gather, and all-to-all. These will be described in detail in the following implementation examples and will not be elaborated upon here.

[0073] An optimizer is the part of a deep learning model (e.g., a large model) that iteratively updates the model parameters through specific computations during the training process.

[0074] Optimizer state, information related to the optimizer, including but not limited to model parameters, gradients, and momentum.

[0075] As the number of model parameters increases, the amount of data generated during the training of large models also increases proportionally. The data generated during the training of large models includes, but is not limited to, at least one of the following: activations generated during forward computation, gradients generated during backward computation, or momentum generated during model parameter updates. This data is stored in memory after generation, occupying a certain amount of memory space.

[0076] Activations are dynamic memory data, and the memory space they occupy is not fixed. Furthermore, after activations are generated during the forward computation of the current training round, they are stored in memory so that they can be read and used during the backward computation of the current training round. Once used, the memory space occupied by the activations is released because the activations from the current training round are no longer needed in the next training round.

[0077] Gradients and momentum are static memory data, and the memory space occupied by gradients and momentum is fixed. Furthermore, after gradients are generated during the backpropagation process in any training round, and momentum is generated during the model parameter update process in any training round, gradients and momentum are stored in memory. The memory space occupied by gradients and momentum will not be released until the large model has converged.

[0078] With a fixed computing power on the GPU, the smaller the memory space occupied by these data, the greater the adjustable space of hyperparameters used in training large models, resulting in higher training efficiency. For example, hyperparameters include the learning rate, which indicates the step size used to update the model parameters, i.e., the magnitude of each parameter update. Furthermore, according to the scaling law, within a certain range, the more model parameters there are, the better the trained large model performs and the higher its predictive performance. Therefore, with a fixed amount of computing power on the GPU, the smaller the memory space occupied by these data, the greater the memory space that model parameters can occupy; a larger number of model parameters leads to better performance of the trained large model.

[0079] Therefore, reducing the memory footprint of data generated during the training of large models, such as activation, gradient, and momentum data, is beneficial for improving the training efficiency and performance of large models. Compression of this data, also known as quantization, reduces the number of bits used to decrease memory usage. As explained above, the training process of a large model involves multiple rounds, each including forward computation, backward computation, and model parameter update. This embodiment of the application can compress data generated in any one or more of these three processes to reduce memory footprint while maintaining the training efficiency and performance of the large model. This model acquisition method can be executed by a chip, including but not limited to the aforementioned computing card. As shown in Figure 2, the chip can be any one of multiple chips in a chip cluster, and these chips can be deployed in one or more computers. The type of large model acquired through this method includes, but is not limited to, transform types. Taking the model acquisition method executed by the chip as an example, and referring to Figure 1, the process of processing the data generated during the forward calculation process, the backward calculation process, and the model parameter update process in this application embodiment will be described in turn.

[0080] The first processing step addresses the multiple activations generated during the forward computation.

[0081] On one hand, the chip determines the number of compressed bits for each of the multiple activations. The number of compressed bits for each activation refers to the number of bits in the compressed activation sequence. If the number of bits for an activation is greater than the number of bits for the compressed activation sequence, then compression of the activation is required. If the number of bits for an activation is equal to the number of bits for the compressed activation sequence, then compression of the activation is not required.

[0082] On the other hand, the chip groups the multiple activations based on their data characteristics, resulting in multiple groups. Each group includes a subset of the activations. Different groups may include different activations from the multiple activations.

[0083] Therefore, based on the number of bits after compression and multiple groups, some or all of the activations in multiple activations can be compressed to obtain multiple compressed activations.

[0084] In one example, the number of bits after compression can be flexibly configured according to actual needs, and this application does not limit the number of bits after compression. For example, the active bit depth is 32 bits, and the compressed bit depth is 16 bits, 8 bits, or 4 bits, etc. Another example is that the active bit depth is 16 bits, and the compressed bit depth is 8 bits or 4 bits, etc.

[0085] For example, when the number of bits is 32, the data format includes, for example, single-precision floating point (FP32). When the number of bits is 16, the data format includes, for example, brain floating point 16 (BF16) or half-precision floating point (FP16). When the number of bits is 8, the data format includes, for example, 8-bit floating point (FP8) or 8-bit integer (INT8). When the number of bits is 4, the data format includes, for example, 4-bit floating point (FP4).

[0086] In another example, the number of bits after compression is determined based on at least one of the following pieces of information.

[0087] The first piece of information is the compression ratio of the activations. The compression ratio refers to the percentage of memory space that can be reduced by compression processing compared to the total memory space occupied by multiple activations. For example, a compression ratio of 25% means that the compressed activations occupy 75% of the total memory space occupied by the multiple activations, reducing memory usage by 25%.

[0088] For example, the higher the compression ratio, the greater the memory space that needs to be reduced, and therefore the smaller the number of bits after compression.

[0089] The second piece of information is the order of the forward computation corresponding to the activation within the forward computation process. The forward computation process includes multiple ordered forward computations, each of which can be used to generate an activation. The forward computation corresponding to the activation is also the forward computation used to generate that activation.

[0090] For example, a larger order of forward computation indicates that the forward computation is a later step in the forward computation process. Later forward computations generate activations that are more sensitive to compression, resulting in a larger number of bits after compression. Conversely, a smaller order of forward computation indicates that the forward computation is an earlier step in the forward computation process. Earlier forward computations generate activations that are less sensitive to compression, resulting in a smaller number of bits after compression. In other words, the number of bits after compression for activations is positively correlated with the order.

[0091] Optionally, when the order is greater than or equal to a sequence threshold, the activation type is deep; when the order is less than the sequence threshold, the activation type is shallow. The compressed bit count of deep activations is greater than or equal to the compressed bit count of shallow activations. Dividing the activation type into two layers (i.e., deep and shallow) is just one example; the activation type can also be divided into three or more layers. In two adjacent layers, the compressed bit count of the deeper activation is greater than or equal to the compressed bit count of the shallower activation.

[0092] The third piece of information is whether the gradient corresponding to the activation is backpropagated during the backpropagation process. As explained above, the gradient can be obtained by backpropagating based on the activation and loss function; therefore, the gradient corresponding to the activation is the gradient obtained by backpropagating based on the activation.

[0093] When the gradient corresponding to the activation propagates back through the reverse computation process, the activation is highly sensitive to compression. In other words, compressing the activation may lead to a large compression error. This is because compressing the activation may introduce compression errors, and these errors also propagate back through the gradient during the reverse computation, causing an accumulation of compression errors and resulting in a large overall compression error. When the gradient corresponding to the activation propagates back through the reverse computation process, the activation type becomes cumulative.

[0094] Referring to Figure 3, the input represents the activation. During forward computation, the activation can enter the Swish-gated linear unit (SwiGLU) to obtain the output. During backward computation, the activation and the output gradient (OGrad) participate in the calculation of the SwiGLU gradient (SwiGLU Grad), and the SwiGLU Grad participates in the calculation of the input gradient (IGrad). IGrad propagates backward during backward computation, thus the compression error of the activation also propagates backward. The activation type is cumulative.

[0095] When the gradient corresponding to the activation propagates back through the backward computation process, the activation is less sensitive to compression. In other words, compressing the activation results in a smaller compression error. This is because while compression may introduce compression error, since the gradient corresponding to the activation does not propagate back through the backward computation process, this compression error does not propagate and therefore does not accumulate, resulting in a smaller compression error. When the gradient corresponding to the activation does not propagate back through the backward computation process, the activation type is non-cumulative.

[0096] Referring to Figure 4, the input represents the activation. During the forward computation, the activation is calculated with the weights using a general matrix multiplication (GEMM) to obtain the output. During the backward computation, IGrad (obtained by GEMM calculation of the weights and OGrad) propagates back through the backward computation process. The activation and OGrad participate in the calculation of the weight gradient (WGrad), but WGrad does not propagate back through the backward computation process. Therefore, the compression error of the activation does not propagate back through the backward computation process, and the activation type is non-cumulative.

[0097] For example, since cumulative activations are more sensitive to compression and non-cumulative activations are less sensitive to compression, the number of bits after compression for cumulative activations is greater than or equal to the number of bits after compression for non-cumulative activations.

[0098] The fourth piece of information is whether the memory space occupied by cumulative activation is the same as that occupied by non-cumulative activation.

[0099] For example, when the memory space occupied by cumulative activation is the same as that occupied by non-cumulative activation, the third piece of information mentioned above is considered when determining the number of bits after compression. When the memory space occupied by cumulative activation is different from that occupied by non-cumulative activation, the third piece of information mentioned above may not be considered when determining the number of bits after compression.

[0100] The above explains four pieces of information that may be considered when determining the number of bits after compression.

[0101] Optionally, in this application embodiment, combining the second and third information, the activation type is divided into the following four types, and the sensitivity of the activation to compression decreases in the following four types: cumulative and deep, cumulative and shallow, non-cumulative and deep, and non-cumulative and shallow.

[0102] When compression of activations is required, i.e., when the compression ratio is not equal to 0%, the number of bits after compression for activations with high sensitivity is greater than or equal to the number of bits after compression for activations with low sensitivity.

[0103] When it is necessary to reduce memory usage, i.e., when the compression ratio needs to be reduced, the number of bits after compression should be reduced first for activations with low sensitivity.

[0104] When it is possible to increase memory usage, i.e., when the compression ratio can be increased, prioritize increasing the number of bits after compression for highly sensitive activations.

[0105] Therefore, the number of activated compressed bits can be dynamically adjusted according to the compression ratio to achieve various compression goals, minimize compression errors, and ensure the training effect of large models.

[0106] As shown in Figure 5, after model initialization, the chip executes the forward computation process. For multiple activations generated during the forward computation, the activation type (cumulative or non-cumulative) is determined, as well as the order of the forward computation corresponding to each activation. Combining the activation type, order, and compression ratio, the number of bits after compression is determined, facilitating subsequent compression processing of some or all activations. Taking the example where the memory space occupied by cumulative and non-cumulative activations is the same, Table 1 provides an illustration. Note that both cumulative and non-cumulative activations have 16 bits before compression.

[0107] Table 1

[0108] When the compression ratio is 0%, no compression is required. Therefore, the number of bits after compression for both cumulative activation and non-cumulative activation is 16 bits, which is the same as the number of bits when uncompressed.

[0109] With a compression ratio of 25%, compression is required to reduce the memory space occupied by 25%. Since the sensitivity of cumulative activation is higher than that of non-cumulative activation, the number of bits after compression of non-cumulative activation is reduced first. That is, non-cumulative activation is selected for compression first, and the number of bits after compression of non-cumulative activation is determined to be 8 bits.

[0110] With a compression ratio of 50%, compression is required to reduce the memory space occupied by 50%. Since the number of bits after compression of non-cumulative activation has been reduced first, the number of bits after compression of cumulative activation will be reduced in this case. That is, cumulative activation will be compressed, and the number of bits after compression of cumulative activation will be determined to be 8 bits.

[0111] With a compression ratio of 62.5%, compression is required to reduce the memory space occupied by 62.5%. Since the sensitivity of cumulative activation is higher than that of non-cumulative activation, the number of bits after compression of non-cumulative activation is reduced first. That is, non-cumulative activation is selected for compression first, and the number of bits after compression of non-cumulative activation is determined to be 4 bits.

[0112] In related technologies, the number of bits after compression is determined based on experience. This method relies on experience and is only applicable to existing calculation types, as users only have experience with existing calculation types, making it quite limited. The embodiments of this application, however, do not rely on experience, because the first to fourth pieces of information mentioned above do not require experience to obtain.

[0113] As mentioned earlier, in addition to determining the number of bits after compression, the chip also groups multiple activations based on their data characteristics, resulting in multiple groups. Each group includes a subset of the activations. The data characteristics of the activations are inherent features of each activation. The grouping of multiple activations based on their inherent features in this embodiment is adaptive and flexible.

[0114] For example, the activated data characteristics include the activated data distribution and differences. Optionally, as shown in FIG6, the chip first groups the multiple activations according to the activated data distribution to obtain multiple groups A, and the differences between the activations included in each group A are the intra-group differences of group A. Then, for each group A, the chip performs a second grouping of group A according to the intra-group differences of group A to obtain one or more groups B. Thus, for multiple groups A, the chip can obtain multiple groups B by performing a second grouping, and the number of groups B is greater than or equal to the number of groups A.

[0115] Optionally, when the data distribution is normally distributed or skewed (including positively skewed and negatively skewed distributions), multiple activations correspond to a mode (also called the mean). As shown in Figure 7, the probability density function of a normally or skewed distribution is a bell-shaped curve, with the horizontal axis representing the value and the vertical axis representing the probability of occurrence. The mode is the value with the highest probability of occurrence, i.e., the value corresponding to the peak of the bell-shaped curve. In the first grouping based on the data distribution, the activations that are the majority and have relatively similar values ​​are grouped into one group A, which can be denoted as the normal group. The other activations besides those in group A are grouped into another group A, which can be denoted as the outlier group. For example, 95% of the activations are grouped into the normal group. The difference between the activations in the normal group and the mode is less than the second reference threshold. The activations in the normal group are those whose values ​​are relatively close to the mode. Five percent of the activations in a group are identified as outliers. The difference between the activations in the outlier group and the mode is greater than or equal to the second reference threshold. The activations in the outlier group are those whose values ​​deviate significantly from the mode.

[0116] Alternatively, when the data distribution is uniform, the chip can sort multiple activations in descending or ascending order of values ​​to obtain sorted activations. During the first grouping based on the data distribution, the sorted activations are grouped into multiple groups A, where different groups A may contain the same or different numbers of activations.

[0117] Regardless of the data distribution from which multiple groups A are obtained, groups A can be regrouped based on the differences within each group to obtain groups B. The number of activations in different groups B can be the same or different. For example, the number of activations in group B obtained by regrouping the normal groups may be different from (e.g., greater than) the number of activations in group B obtained by regrouping the abnormal groups. Optionally, the difference between the activations in group B is less than a first threshold. That is, the activation values ​​in group B are relatively close, resulting in a smaller compression error.

[0118] For example, in embodiments of this application, the compressed bit depth determined above can be updated based on the difference between the activations included in group B to obtain an updated compressed bit depth. When compressing multiple activations subsequently, compression is performed according to the updated compressed bit depth. Optionally, the larger the difference between the activations included in group B, the larger the updated compressed bit depth, and the updated compressed bit depth is positively correlated with the difference between the activations included in group B.

[0119] For example, if the difference between activations in group B is less than the first threshold and less than the third reference threshold (the third reference threshold is less than the first threshold), that is, the intra-group difference of group B is small enough (denoted as type 1), then the updated compressed bit length is equal to the compressed bit length determined above (that is, there is no need to update the compressed bit length).

[0120] For example, if the difference between activations in group B is less than the first threshold and greater than or equal to the third reference threshold, meaning the intra-group difference in group B is large (denoted as type 2), then the updated compressed bit depth is greater than the compressed bit depth determined above, i.e., the compressed bit depth is increased, resulting in the updated compressed bit depth. Here, the updated compressed bit depth is less than or equal to the number of activation bits when no compression is applied; less than indicates that compression is required, and equal to indicates that compression is not required.

[0121] Optionally, the updated compressed bit length corresponding to group B of type 2 is greater than the updated compressed bit length corresponding to group B of type 1.

[0122] Taking the normal distribution of multiple activations as an example, as shown in Figure 8, the first grouping is performed according to the normal distribution to obtain the normal group and the abnormal group. The normal group includes most activations with relatively similar values, while the abnormal group includes a small number of activations with excessively large or small values.

[0123] For the regular groups, a second grouping is performed based on the intra-group differences, resulting in multiple groups B, where the intra-group difference for each group B is less than the first threshold. Based on this, if the intra-group difference of group B is sufficiently small and belongs to type 1 above, the activations included in group B are compressed according to the compressed bit depth. If the intra-group difference of group B is large and belongs to type 2 above, the compressed bit depth is increased, and the activations included in group B are compressed according to the updated compressed bit depth, or the activations included in group B are not compressed, to reduce compression error and ensure compression accuracy. For example, if the intra-group difference of group B is greater than the second threshold, the activations included in group B are not compressed.

[0124] For the abnormal group, a second grouping is performed based on the difference within the group, which will not be elaborated here. Since the number of activations in the abnormal group differs from that in the regular group, the group size used for the second grouping can be different.

[0125] With the updated compressed bit depth for each group B already determined, compression can be performed on a per-group-B basis. If the updated compressed bit depth for a group B is equal to the number of active bits when the activation is not compressed, then no compression is needed for the activations included in group B. If the updated compressed bit depth for a group B is less than the number of active bits when the activation is not compressed, then compression is needed for the activations included in group B, and the number of active bits after compression is equal to the updated compressed bit depth. The updated compressed bit depth can be 8 bits (e.g., FP8 or INT8) or 4 bits (e.g., FP4), etc., and this embodiment does not limit this.

[0126] Therefore, the chip compresses some or all of the activations from multiple activations, resulting in multiple compressed activations. These compressed activations can be stored in memory. During subsequent backpropagation, the chip can read these compressed activations from memory, decompress them to obtain multiple restored activations, and then perform backpropagation based on these restored activations and the loss function to obtain the gradient.

[0127] In related technologies, multiple activations are grouped according to their generation order during the forward computation process, with each group containing the same number of activations. Alternatively, in related technologies, multiple activations are sorted according to their generation order and then grouped according to a fixed group size (i.e., the number of activations in each group). However, sorting multiple activations according to their generation order may result in a disordered arrangement of the sorted activations. Grouping according to a fixed group size may lead to significant differences in the values ​​of activations within a group, causing substantial compression errors. This application's embodiment adaptively groups activations based on their inherent characteristics, avoiding large differences in activation values ​​within a group and reducing compression errors.

[0128] The second processing step deals with the multiple gradients generated during the reverse computation process.

[0129] The chip groups multiple gradients based on their data characteristics (e.g., data distribution and differences), resulting in multiple groups. Each group includes a subset of gradients, and different groups may contain different gradients. The chip can flexibly configure the number of bits after compression for multiple gradients according to actual needs. Based on the number of bits after compression and the multiple groups, it compresses some or all of the gradients to obtain multiple compressed gradients.

[0130] The method by which the chip groups multiple gradients and obtains multiple compressed gradients can be found in the explanation of multiple activations in the first processing step, and will not be repeated here.

[0131] The chip can be any one of the multiple chips in a chip cluster. When the current chip obtains multiple compressed gradients, the current chip can communicate with other chips in a group. The gradients transmitted during the group communication process can be compressed gradients. The compressed gradients are low-precision format data with a small data volume, which can shorten the transmission time and improve training efficiency.

[0132] In this embodiment, the current chip can receive multiple compressed gradients. It then decompresses the multiple compressed gradients obtained by the current chip itself and the received gradients, and sums the decompressed gradients. The decompressed gradients are high-precision format data. This ensures that gradients transmitted between different chips are in low-precision format, while the current chip uses high-precision format data for calculations. This avoids using low-precision format data during calculations (e.g., addition), improving training efficiency while maintaining computational accuracy, making the training process of large models lossless. For example, as shown in Figure 9, during addition calculations, the transmitted gradients g1 to gn are in low-precision format. g1 to gn are decompressed to obtain high-precision format data, and then addition is performed on this high-precision format data to obtain the result g, which is also in high-precision format.

[0133] In related technologies, low-precision format data is directly used for addition calculations. However, the numerical representation range of low-precision format data is relatively limited, and the addition result of different low-precision format data may result in numerical overflow. For example, the upper limit of the numerical representation range of 8-bit low-precision format data is 127. If one low-precision format data represents 120 and another low-precision format data represents 10, then the sum of these two low-precision format data is 130, which exceeds the upper limit of 127, resulting in numerical overflow. To address this, related technologies maintain a scaling factor s for each piece of low-precision format data. During addition calculations, the low-precision format data is first divided by the scaling factor s to obtain updated low-precision format data, and then the addition calculation is performed on the updated low-precision format data to avoid numerical overflow. For example, referring to Figure 10, during addition calculations, the transmitted gradients g1 to gn are all in low-precision format. g1 to gn are divided by scaling factors s1 to sn respectively to obtain updated gradients, which are also in low-precision format. Then, addition calculations and decompression (also called dequantization) are performed on the updated low-precision format data to obtain the result g, which is in high-precision format data. However, the scaling factors are only updated every 1000 training steps, or only when numerical overflow has already occurred. Untimely updates can still lead to numerical overflow problems, affecting training accuracy and even causing training failure (e.g., inability to converge). The embodiments of this application first decompress the low-precision format data into high-precision format data before performing addition calculations, which not only completely avoids numerical overflow problems but also ensures training accuracy.

[0134] As mentioned above, the chip can be any one of the multiple chips included in the chip cluster. Taking the four chips NPU1, NPU2, NPU3 and NPU4 as an example, the addition calculation in the related technology and the addition calculation in the embodiment of this application will be compared and explained.

[0135] A related technology provides a set communication method called reduce scatter, which involves additive computation of low-precision format data. In a four-chip architecture, each chip divides multiple locally acquired compressed gradients into four data parts, each part containing one-quarter of the compressed gradients. Each chip sends one data part to the other three chips (sending three out of four parts in total) and receives data sent by the other three chips (receiving three data parts in total). Both the sent and received data are compressed low-precision format data. Then, the four chips perform additive computation on the low-precision format data to achieve reduce scatter. For example, referring to Figure 11, the four data parts of NPU1 include A0 to A3, the four data parts of NPU2 include B0 to B3, the four data parts of NPU3 include C0 to C3, and the four data parts of NPU4 include D0 to D3. NPU1 sends A1 to NPU2 and receives B0 from NPU2. NPU1 sends A2 to NPU3 and receives C0 from NPU3. NPU1 sends A3 to NPU4 and receives D0 from NPU4. NPU1 performs addition on A0, B0, C0, and D0 (all low-precision format data), and decompresses the resulting A0+B0+C0+D0 to obtain A0'+B0'+C0'+D0', thus achieving reduce scatter. Similarly, NPU2 obtains A1+B1+C1+D1 and decompresses it to obtain A1'+B1'+C1'+D1', NPU3 obtains A2+B2+C2+D2 and decompresses it to obtain A2'+B2'+C2'+D2', and NPU4 obtains A3+B3+C3+D3 and decompresses it to obtain A3'+B3'+C3'+D3'. Further details are omitted here.

[0136] In this embodiment, while avoiding the addition calculation of low-precision format data, an equivalent of reduce scatter is achieved. For example, reduce scatter is replaced by all-to-all + decompression + addition calculation of high-precision format data. All-to-all means that each chip receives low-precision format data sent by other chips without performing calculations. After receiving the data, it first decompresses the low-precision format data to restore it to high-precision format data, and then performs addition calculations on the high-precision format data. For example, referring to Figure 12, NPU1 sends A1 to NPU2 and receives B0 from NPU2; NPU1 sends A2 to NPU3 and receives C0 from NPU3; NPU1 sends A3 to NPU4 and receives D0 from NPU4. NPU1 decompresses A0, B0, C0, and D0 respectively to obtain A0', B0', C0', and D0'. Then, it performs addition on A0', B0', C0', and D0' to obtain A0'+B0'+C0'+D0', which is equivalent to the reduce scatter mentioned above. Similarly, NPU2 obtains A1'+B1'+C1'+D1', NPU3 obtains A2'+B2'+C2'+D2', and NPU4 obtains A3'+B3'+C3'+D3', which will not be elaborated here.

[0137] Another related technology provides an all-reduce communication method, which also involves addition calculations of low-precision format data. In this method, each of the four chips locally obtains multiple compressed gradients. Each chip sends all its locally obtained data to the other three chips and receives all data sent by the other three chips. Both the sent and received data are compressed low-precision format data. Then, the four chips perform addition calculations on the low-precision format data to achieve all-reduce. For example, referring to Figure 13, NPU1 sends A0 to A3 to NPU2, NPU3, and NPU4 respectively. NPU1 receives B0 to B3 from NPU2, C0 to C3 from NPU3, and D0 to D3 from NPU4. NPU1 performs addition and decompression on A0 to A3, B0 to B3, C0 to C3, and D0 to D3 (all low-precision format data) to obtain A0'+B0'+C0'+D0', A1'+B1'+C1'+D1', A2'+B2'+C2'+D2', and A3'+B3'+C3'+D3', thus achieving all reduce. The implementation of all reduce on NPU2, NPU3, and NPU4 is similar to that on NPU1, and will not be elaborated here.

[0138] In this embodiment, while avoiding the addition calculation of low-precision format data, the equivalent of all reduce is achieved. For example, all reduce is replaced by all to all + decompression + addition calculation of high-precision format data + all gather. For example, referring to Figure 14, NPU1 obtains A0'+B0'+C0'+D0' (the process is detailed in the description corresponding to Figure 12, and will not be repeated here) and then performs all gather, that is, NPU2, NPU3, and NPU4 send A0'+B0'+C0'+D0' respectively. Similarly, NPU2 obtains A1'+B1'+C1'+D1' and sends it to NPU1, NPU3, and NPU4 respectively; NPU3 obtains A2'+B2'+C2'+D2' and sends it to NPU1, NPU2, and NPU4 respectively; NPU4 obtains A3'+B3'+C3'+D3' and sends it to NPU1, NPU2, and NPU3 respectively, thus being equivalent to the above-mentioned all reduce.

[0139] The above text uses the example of ensemble communication between different chips based on compressed gradients to illustrate the concept. Different chips can also communicate based on other data (such as compressed activation data), which is based on the same principle as ensemble communication based on compressed gradients. This will not be elaborated on here.

[0140] The third processing step addresses the momentum generated during the model parameter update process.

[0141] The chip groups multiple momentum units based on their data characteristics (e.g., data distribution and differences), resulting in multiple groups. Each group includes a portion of the momentum units, and different groups may include different momentum units. The chip can flexibly configure the number of bits after compression for the multiple momentum units according to actual needs. Based on the number of bits after compression and the multiple groups, some or all of the momentum units are compressed to obtain multiple compressed momentum units.

[0142] The method by which the chip groups multiple momentum and obtains multiple compressed momentum can be found in the explanation of multiple activations in the first processing step, and will not be repeated here.

[0143] As shown in Figure 15, gradients are obtained through forward and backward computation (including forward and backward computation). These gradients are then adaptively grouped and compressed to obtain multiple compressed gradients. The optimizer decompresses these compressed gradients to obtain multiple decompressed gradients. Based on these decompressed gradients, momentum is determined (for the model parameter update process included in the first round of training). This momentum is used to update the model parameters. Furthermore, the momentum is adaptively grouped and compressed to obtain multiple compressed momentum. In subsequent rounds of training, including model parameter updates, the optimizer can decompress these compressed momentums to obtain multiple decompressed momentum. These decompressed momentums participate in momentum updates to obtain updated momentum, which is then used to update the model parameters.

[0144] The situation shown in Figure 15 is only an example. In the process of updating model parameters, momentum can also be omitted. In this case, the optimizer can decompress the compressed gradient and update the model parameters based on the decompressed gradient.

[0145] In related technologies, a zero-offload approach is proposed based on the zero method. The zero method refers to storing optimizer states such as gradients and momentum on different chips (i.e., computing cards such as NPUs or GPUs). The zero-offload approach is a heterogeneous computing method that enables collaboration between chips and the central processing unit (CPU). In this approach, the forward and backward computation processes are executed by the chip to obtain gradients, which the chip then sends to the CPU. Based on the received gradients and the CPU's local momentum, the CPU performs a model parameter update process through the Adam optimizer to obtain updated model parameters. The CPU then sends the updated model parameters back to the chip so that the chip can perform the forward and backward computation processes in the next round of training. To improve training efficiency, this related technology employs parallel computation and communication between the chip and the CPU. For example, after the chip performs backward computation and obtains a portion of the gradients, it sends them to the CPU. While sending the obtained gradients, it continues to perform backward computation, so that by the end of the backward computation process, the CPU has received most of the gradients. For example, in related technologies, after the CPU executes model parameter updates and obtains partially updated model parameters, it sends them to the chip. While sending the obtained updated model parameters, it continues to perform model parameter updates. Therefore, by the end of the model parameter update process, the chip has received most of the updated model parameters. However, the model parameter update process in related technologies requires the transmission of gradients and updated model parameters between the chip and the CPU. The communication overhead between the chip and the CPU can easily become a performance bottleneck. When using a parallel computation and communication approach, the implementation complexity is high, requiring the introduction of additional hyperparameters and manual debugging. This results in low usability of the methods provided by related technologies and a strong dependence on the Adam optimizer on the CPU, leading to significant limitations. In the embodiments of this application, the model parameter update process can be implemented directly through the chip, without relying on the CPU or being limited to the Adam optimizer. It is simple and easy to implement, requiring no additional hyperparameters and no manual debugging.

[0146] The foregoing described the processing of data generated during the forward computation, backward computation, and model parameter update processes in this application's embodiments. These processes enable the training and acquisition of a large model. For example, the large model could be a text generation model used to generate and output response text from input text. The methods provided in this application's embodiments improve the training efficiency and accuracy of the text generation model, ensuring its training effectiveness. Therefore, when using the text generation model, it can generate and output accurate and reasonable response text from input text, guaranteeing a positive user experience.

[0147] In addition, taking the training of LLaMA3-8B (an open-source large model) through the method provided in the embodiments of this application as an example, as shown in Table 2, without compression, the memory usage is 38 gigabytes (GB), and with compression, approximately 50% memory reduction can be achieved.

[0148] Table 2

[0149] As shown in Figure 16, the training loss when the activation is compressed to 8 bits (the compressed bit depth of the activation is 8 bits) using the method provided in this embodiment can be aligned with the baseline training loss. In Figure 16, the horizontal axis represents the number of training steps, and the vertical axis represents the training loss. Based on this, as shown in Figure 17, the training loss when the gradient and momentum are further compressed can also be aligned with the baseline training loss. In Figure 17, the horizontal axis represents the number of training steps, and the vertical axis represents the training loss. `adam32bit_grad16` represents a compressed momentum bit depth of 32 bits and a compressed gradient bit depth of 16 bits; `adam32bit_grad32` represents a compressed momentum bit depth of 32 bits and a compressed gradient bit depth of 32 bits; `adam8bit_grad32` represents a compressed momentum bit depth of 8 bits and a compressed gradient bit depth of 32 bits; and `adamW4bit_grad32` represents a compressed momentum bit depth of 4 bits and a compressed gradient bit depth of 32 bits.

[0150] Therefore, the method provided in this application embodiment can reduce memory usage while maintaining training accuracy, thus ensuring that large models have good training results.

[0151] Based on the above description, referring to Figure 18, the model acquisition method provided in this application embodiment includes the following steps 1801 to 1804.

[0152] Step 1801: Obtain multiple first data points, which are data generated during the first training process of the large model.

[0153] Step 1802: Obtain multiple first groups based on the data characteristics of multiple first data, with each first group including a portion of the first data from the multiple first data.

[0154] Step 1803: Compress the first data included in the multiple first groups to obtain multiple second data.

[0155] Step 1804: Perform a second training process for the large model based on multiple second data sets.

[0156] Among them, multiple first data include at least one of activation, gradient, or momentum.

[0157] For example, the first training process is the forward computation process in the current round of training, with multiple first data points as activations and multiple second data points as compressed activations. The second training process is the backward computation process in the current round of training.

[0158] For example, the first training process is the reverse calculation process of the current training process, with multiple first data points as gradients and multiple second data points as compressed gradients. The second training process is the model parameter update process of the current training process.

[0159] For example, the first training process is the model parameter update process in the current round of training, where multiple first data points are momentum and multiple second data points are compressed momentum. The second training process is the model parameter update process in the next round of training.

[0160] In an exemplary embodiment, the difference between the first data included in each first group is less than a first threshold. Each first group corresponds to group B as described above.

[0161] For example, the number of bits of the second data of the first type is greater than the number of bits of the second data of the second type, and the intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group. Wherein, the first group corresponding to the second data of the first type corresponds to group B of the above-mentioned type 2, and the second data of the second type corresponds to group B of the above-mentioned type 1.

[0162] Optionally, the first training process includes a forward computation process, the second training process includes a backward computation process, multiple first data points include activations, and multiple second data points are used to obtain the corresponding gradients during the backward computation process. The number of bits for the third type of second data is greater than or equal to the number of bits for the fourth type of second data, and the gradient corresponding to the third type of second data is backpropagated during the backward computation process. Optionally, the gradient corresponding to the fourth type of second data is not backpropagated during the backward computation process. The third type of second data is obtained by compressing cumulative activations, and the fourth type of second data is obtained by compressing non-cumulative activations.

[0163] Optionally, the first training process includes a forward computation process, which comprises multiple ordered forward computations, and the multiple first data include activations. The number of bits in the second data of the fifth type is greater than or equal to the number of bits in the second data of the sixth type, and the order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type. The order is the order of the forward computation used to generate the first data in the forward computation process. Specifically, the second data of the fifth type is obtained by compressing deep activations, and the second data of the sixth type is obtained by compressing shallow activations.

[0164] In an exemplary embodiment, the method further includes: receiving a plurality of third data, wherein the plurality of third data are data obtained by compression processing; decompressing the plurality of second data and the plurality of third data; and summing the decompressed data. For details on this, please refer to the description of set communication in the second processing step above, which will not be repeated here.

[0165] For example, the method further includes: obtaining at least one second group based on data features of a plurality of first data, each first group including a portion of the first data from the plurality of first data, wherein the difference between the first data included in the second group is greater than a second threshold. Performing a second training process for a large model based on the plurality of second data includes: performing a second training process based on the plurality of second data and the first data included in the second group. For example, the second group corresponds to group B in the above description, where the intra-group difference is greater than the second threshold and no compression is required.

[0166] The above describes the model acquisition method provided by the embodiments of this application. Corresponding to the above method, the embodiments of this application also provide a model acquisition device. This device is applied to any one of the multiple chips included in a chip cluster. The device is used to execute the steps performed by the chip in the method embodiment shown in Figure 18 through the various modules shown in Figure 19. As shown in Figure 19, the model acquisition device provided by the embodiments of this application includes the following modules.

[0167] The acquisition module 1901 is used to acquire multiple first data points, which are data generated during the first training process of the large model.

[0168] The acquisition module 1901 is also used to acquire multiple first groups based on the data characteristics of multiple first data, each first group including a portion of the first data in the multiple first data;

[0169] Processing module 1902 is used to compress the first data included in the first group respectively to obtain the second data;

[0170] Execution module 1903 is used to perform a second training process for the large model based on multiple second data.

[0171] In one possible implementation, the difference between the first data included in each first group is less than a first threshold.

[0172] In one possible implementation, the number of bits of the second data of the first type is greater than the number of bits of the second data of the second type, and the intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group.

[0173] In one possible implementation, the first training process includes a forward computation process, the second training process includes a backward computation process, multiple first data points include activations, and multiple second data points are used to obtain the corresponding gradients during the backward computation process; the number of bits of the third type of second data is greater than or equal to the number of bits of the fourth type of second data, and the gradient corresponding to the third type of second data is backpropagated during the backward computation process. Optionally, the gradient corresponding to the fourth type of second data is not backpropagated during the backward computation process.

[0174] In one possible implementation, the first training process includes a forward computation process, which includes multiple ordered forward computations, and multiple first data including activations; the number of bits of the second data of the fifth type is greater than or equal to the number of bits of the second data of the sixth type, and the order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type, and the order is the order of the forward computations used to generate the first data in the forward computation process.

[0175] In one possible implementation, the acquisition module 1901 is further configured to receive multiple third data, which are data obtained by compression processing; the processing module 1902 is further configured to decompress the multiple second data and the multiple third data, and sum the decompressed data.

[0176] In one possible implementation, the acquisition module 1901 is further configured to acquire at least one second group based on the data features of the multiple first data, each second group including a portion of the first data from the multiple first data, and the difference between the first data included in the second group is greater than a second threshold; the execution module 1903 is further configured to perform a second training process based on the multiple second data and the first data included in the second group.

[0177] In one possible implementation, the multiple first data include at least one of activation, gradient, or momentum.

[0178] It should be understood that the beneficial effects of the device shown in Figure 19 in implementing its functions are the same as those of the method embodiment shown in Figure 18. The device shown in Figure 19 is only illustrated by the division of the above-mentioned functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0179] This application also provides a computer device, which includes a memory and a processor. The memory stores at least one computer instruction, which is loaded and executed by the processor to cause the computer device to perform the model acquisition method shown in FIG18.

[0180] Referring to Figure 20, which shows a schematic diagram of the structure of an exemplary model acquisition device 2000 of this application, the model acquisition device 2000 includes at least one processor 2001, a memory 2003 and at least one network interface 2004.

[0181] The processor 2001 is, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs), programmable logic devices (PLDs), other general-purpose processors or other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, or any combination thereof used to implement the scheme of this application. A PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor. It is worth noting that the processor can be a processor supporting an advanced reduced instruction set machine (RISC) machine (ARM) architecture. It can implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0182] Optionally, the model acquisition device 2000 also includes a bus 2002. The bus 2002 is used to transmit information between the components of the model acquisition device 2000. The bus 2002 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 2002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one line is used in Figure 20, but this does not mean that there is only one bus or one type of bus.

[0183] The memory 2003 may be, for example, volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache.

[0184] By way of example, but not limitation, many forms of ROM and RAM are available. For example, ROM is a compact disc read-only memory (CD-ROM). RAM includes, but is not limited to, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0185] The memory 2003 can also be other types of storage devices capable of storing static information and instructions. Alternatively, it can be other types of dynamic storage devices capable of storing information and instructions. It can also be other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory 2003 may exist independently and be connected to the processor 2001 via bus 2002. The memory 2003 may also be integrated with the processor 2001.

[0186] Network interface 2004 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area network (WLAN). Network interface 2004 can include wired network interfaces and wireless network interfaces. Specifically, network interface 2004 can be an Ethernet interface, such as Fast Ethernet (FE), Gigabit Ethernet (GE), Asynchronous Transfer Mode (ATM), WLAN, cellular network, or combinations thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In some embodiments of this application, network interface 2004 can be used for model acquisition device 2000 to communicate with other devices. Optionally, model acquisition device 2000 includes multiple network interfaces; for example, as shown in FIG20, in addition to network interface 2004, model acquisition device 2000 also includes network interface 2006. Network interface 2006 may be the same type as network interface 2004 or different. This application embodiment does not limit the number and type of network interfaces included in the model acquisition device 2000.

[0187] In specific implementations, as some embodiments, processor 2001 may include one or more NPUs, such as NPU0 and NPU1 as shown in FIG20. Each of these processors may be a single-core processor or a multi-core processor. Here, processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0188] In specific implementations, as some embodiments, the model acquisition device 2000 may include multiple processors, such as processor 2001 and processor 2005 as shown in FIG20. Each of these processors may be a single-core processor or a multi-core processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). For example, processor 2005 as shown in FIG20 includes GPU0 and GPU1.

[0189] In some embodiments, the memory 2003 is used to store program instructions 2010 for executing the scheme of this application, and the processor 2001 can execute the program instructions 2010 stored in the memory 2003. That is, the model acquisition device 2000 can implement the method provided in the method embodiment, i.e., the method shown in FIG18, through the processor 2001 and the program instructions 2010 in the memory 2003. The program instructions 2010 may include one or more software modules. Optionally, the processor 2001 itself may also store program instructions for executing the scheme of this application.

[0190] In specific implementation, the model acquisition device 2000 of this application can correspond to the first device for executing the above method. The processor 2001 in the model acquisition device 2000 reads the instructions in the memory 2003, so that the model acquisition device 2000 shown in FIG20 can execute all or part of the steps in the method embodiment.

[0191] The model acquisition device 2000 can also correspond to the device shown in Figure 19 above, where each functional module is implemented in software using the model acquisition device 2000. In other words, the functional modules included in the device shown in Figure 19 are generated by the processor 2001 of the model acquisition device 2000 reading the program instructions 2010 stored in the memory 2003.

[0192] In the method shown in Figure 18, each step is completed through the integrated logic circuitry of the hardware or software instructions in the processor of the model acquisition device 2000. The steps of the method embodiments disclosed in this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since the storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method embodiments; to avoid repetition, these will not be described again here.

[0193] This application also provides a chip, which includes a processor and an interface circuit. The interface circuit provides program instructions to the processor, and the processor executes the program instructions, so that a computer with the chip installed performs the model acquisition method shown in FIG18.

[0194] This application embodiment also provides another chip, which includes an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the computer with the chip installed executes the model acquisition method shown in Figure 18.

[0195] This application also provides a computer-readable storage medium that stores a program or instructions. When the program or instructions are run on a computer, the computer executes the model acquisition method shown in FIG18.

[0196] This application also provides a computer program or computer program product, which includes computer program code. When the computer program code is run by a computer, the computer executes the model acquisition method shown in FIG18.

[0197] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to this application are generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0198] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0199] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0200] In this application, the term "at least one" means one or more, and the term "multiple" means two or more; for example, "multiple" means two or more. The terms "system" and "network" are often used interchangeably in this document.

[0201] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing the particular examples only and is not intended to be limiting. As used in the description of the various examples and in the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0202] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0203] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0204] The above are merely embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A model acquisition method characterized by comprising: The method includes: Acquire multiple first data points, which are data generated during the first training process of the large model; Multiple first groups are obtained based on the data characteristics of the multiple first data, and each first group includes a portion of the first data from the multiple first data; The first data included in the plurality of first groups are compressed to obtain a plurality of second data; The second training process of the large model is performed based on the multiple second data.

2. The method of claim 1, wherein, The difference between the first data in each first group is less than the first threshold.

3. The method according to claim 1 or 2, characterized in that, The number of bits of the second data of the first type is greater than the number of bits of the second data of the second type. The intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group.

4. The method according to any one of claims 1-3, characterized in that, The first training process includes a forward computation process, the second training process includes a backward computation process, the plurality of first data includes activations, and the plurality of second data is used to obtain corresponding gradients in the backward computation process; The number of bits of the second data of the third type is greater than or equal to the number of bits of the second data of the fourth type, and the gradient corresponding to the second data of the third type is backpropagated during the back calculation process.

5. The method according to any one of claims 1-4, characterized in that, The first training process includes a forward computation process, which includes multiple ordered forward computations, and the multiple first data includes activations; The number of bits of the second data of the fifth type is greater than or equal to the number of bits of the second data of the sixth type. The order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type. The order is the order of the forward computation used to generate the first data in the forward computation process.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Receive multiple third data, wherein the multiple third data are data obtained by compression processing; The plurality of second data and the plurality of third data are decompressed, and the decompressed data is summed.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: At least one second group is obtained based on the data characteristics of the plurality of first data, each second group includes a portion of the first data in the plurality of first data, and the difference between the first data included in the second group is greater than a second threshold. The second training process of the large model based on the plurality of second data includes: The second training process is performed based on the plurality of second data and the first data included in the second group.

8. The method according to any one of claims 1-7, characterized in that, The plurality of first data includes at least one of activation, gradient, or momentum.

9. A model acquisition apparatus characterized by comprising: The device includes: The acquisition module is used to acquire multiple first data points, which are data generated during the first training process of the large model. The acquisition module is further configured to acquire multiple first groups based on the data characteristics of the multiple first data, each first group including a portion of the first data in the multiple first data; The processing module is used to compress the first data included in the plurality of first groups respectively to obtain a plurality of second data; An execution module is used to perform a second training process of the large model based on the plurality of second data.

10. The apparatus of claim 9, wherein, The difference between the first data in each first group is less than the first threshold.

11. The apparatus of claim 9 or 10, wherein, The number of bits of the second data of the first type is greater than the number of bits of the second data of the second type. The intra-group difference of the first group corresponding to the second data of the first type is greater than the intra-group difference of the first group corresponding to the second data of the second type. The intra-group difference of the first group is the difference between the first data included in the first group.

12. The apparatus of any one of claims 9-11, wherein, The first training process includes a forward computation process, the second training process includes a backward computation process, the plurality of first data includes activations, and the plurality of second data is used to obtain corresponding gradients in the backward computation process; The number of bits of the second data of the third type is greater than or equal to the number of bits of the second data of the fourth type, and the gradient corresponding to the second data of the third type is backpropagated during the back calculation process.

13. The apparatus of any one of claims 9-12, wherein, The first training process includes a forward computation process, which includes multiple ordered forward computations, and the multiple first data includes activations; The number of bits of the second data of the fifth type is greater than or equal to the number of bits of the second data of the sixth type. The order of the first data in the first group corresponding to the second data of the fifth type is greater than the order of the first data in the first group corresponding to the second data of the sixth type. The order is the order of the forward computation used to generate the first data in the forward computation process.

14. The apparatus according to any one of claims 9-13, characterized in that, The acquisition module is further configured to receive multiple third data, which are data obtained by compression processing; The processing module is further configured to decompress the plurality of second data and the plurality of third data, and to sum the decompressed data.

15. The apparatus according to any one of claims 9-14, characterized in that, The acquisition module is further configured to acquire at least one second group based on the data characteristics of the plurality of first data, each second group including a portion of the first data in the plurality of first data, and the difference between the first data included in the second group is greater than a second threshold. The execution module is used to perform the second training process based on the plurality of second data and the first data included in the second group.

16. The apparatus of any one of claims 9-15, wherein, The plurality of first data includes at least one of activation, gradient, or momentum.

17. A chip, characterized by The chip includes a processor and an interface circuit. The interface circuit provides program instructions to the processor, which executes the program instructions to cause a computer equipped with the chip to perform the model acquisition method according to any one of claims 1-8.

Images