Large-language-model-based long-text generation method capable of realizing context compression
By encoding and decoding long texts and combining a two-stage training strategy to optimize the text compression model, the problem of high resource consumption in long context text processing by large language models is solved, and efficient text compression and generation are achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- INST OF AUTOMATION CHINESE ACAD OF SCI
- Filing Date
- 2024-12-24
- Publication Date
- 2026-05-15
AI Technical Summary
Existing Transformer-based language models require enormous computational and data storage resources when processing long context texts, limiting their application in long contexts, especially in vertical domains such as legal and regulatory Q&A.
A context compression method based on a large language model is adopted. The text to be compressed is encoded to generate a compressed vector, which is then combined with the prompt word embedding vector for autoregressive decoding and mapped to compressed context text. A two-stage training strategy is used to optimize the text compression model.
It reduces the consumption of model computing and data storage resources, and improves the efficiency and effectiveness of large language models in long text processing tasks.
Smart Images
Figure CN2024141898_15052026_PF_FP_ABST
Abstract
Description
A method for generating long texts based on context compression of a large language model
[0001] Cross-reference to related applications
[0002] This application claims priority to Chinese patent application No. 202411564900.9, filed on November 5, 2024, entitled “A method for generating long text based on context compression of a large language model”, which is incorporated herein by reference in its entirety. Technical Field
[0003] This application relates to the field of artificial intelligence technology, and in particular to a method for generating long texts based on context compression of a large language model. Background Technology
[0004] With the success of chat language models like ChatGPT, Transformer-based language models are playing an increasingly important role in human life, such as question answering, copywriting, and code generation. However, due to the inherent limitations of current Transformer-based language models, they need to retain all contextual information in memory. This results in significant computational and storage resource consumption when processing texts with long contexts, limiting the application of language models in long contexts. In vertical domain applications, language models often need to handle large amounts of specialized knowledge text. For example, in some scenarios, models need to perform text-based question answering based on numerous laws and regulations, the length of which often far exceeds the input context window size of the language model. Therefore, existing language models struggle to complete such tasks or require enormous computational and data storage resources. Summary of the Invention
[0005] This application provides a long text generation method based on large language model context compression, which solves the technical problem that large language models consume huge model computing resources and data storage resources when processing long context texts in the prior art.
[0006] This application provides a long text generation method based on context compression of a large language model, the method including the following steps:
[0007] Obtain the context text to be compressed and the prompt word text, and perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compression vector. The context text to be compressed is the text used by the large language model to perform text processing tasks.
[0008] The compressed vector is concatenated with the prompt word embedding vector corresponding to the prompt word text, and the concatenated fusion feature is subjected to autoregressive decoding processing to obtain multiple corresponding target token identifiers.
[0009] According to a preset vocabulary, the target token identifiers are mapped one by one to text strings, and the text strings are combined into compressed context text.
[0010] In some embodiments, performing compression-based encoding on the context text to be compressed to obtain a corresponding compression vector includes:
[0011] Each text character of the context text to be compressed is converted into a token identifier to obtain the corresponding token list;
[0012] The token list is divided according to a preset sequence length, and the resulting sub-sequences are stored sequentially in a queue.
[0013] Subsequences are selected from the queue and input into an encoder network for compression for encoding processing. The vectors obtained from the encoding processing are then concatenated to obtain a concatenated vector.
[0014] The concatenated vector is input into a multilayer perceptron for feature dimension mapping to obtain the corresponding compressed vector.
[0015] In some embodiments, the autoregressive decoding process performed on the concatenated fused features to obtain multiple corresponding target token identifiers includes:
[0016] Get the current cumulative prediction token identifier;
[0017] The fused features obtained by splicing are spliced with the cumulative embedding vector corresponding to the cumulative predicted token identifier, and the spliced features are input into the decoding model for autoregressive decoding to obtain the corresponding target token identifier.
[0018] If the current target token identifier is a terminator, stop autoregressive decoding; if the current target token identifier is not a terminator, add the current target token identifier as the cumulative predicted token identifier.
[0019] In some embodiments, the compressed context text is obtained by calling a text compression model, which includes a text compression unit model, a compression vector mapping model, and a compression vector decoding model. The training method of the text compression model includes:
[0020] Obtain the context text training dataset, and divide the context text training dataset into a short text dataset and a long text dataset according to a preset text length threshold;
[0021] The text compression model is trained in the first stage using the short text dataset to obtain a first text compression model. During the first stage of training, the parameters of the text compression unit model, the compression vector mapping model, and the compression vector decoding model in the first text compression model are updated.
[0022] The first text compression model is trained in the second stage using the long text dataset to obtain the second text compression model. During the second stage training, the parameters of the text compression unit model in the first text compression model are frozen, and the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated.
[0023] In some embodiments, the short text dataset includes short text samples, first prompt word samples, and first token prompt symbol samples. The step of training the text compression model using the short text dataset in the first stage to obtain a first text compression model includes:
[0024] The short text sample is input into the text compression unit model of the text compression model for encoding processing to obtain the corresponding first compression vector. The first compression vector is then input into the compression vector mapping model of the text compression model for feature mapping to obtain the text compression vector.
[0025] Determine the first embedding vector of the first prompt word sample and the second embedding vector of the first token prompt sample;
[0026] The text compression vector, the first embedding vector, and the second embedding vector are concatenated along the sequence direction to obtain the first concatenation feature;
[0027] The first concatenated feature is input into the compression vector decoding model of the text compression model for decoding processing to obtain the token identifier of the first text to be predicted;
[0028] Based on the token identifier of the first text to be predicted and the first token prompt sample, construct the first cross-entropy loss function;
[0029] The first text compression model is obtained by backpropagating through the first cross-entropy loss function in the text compression model and updating the parameters of the text compression unit model, compression vector mapping model and compression vector decoding model in the text compression model during the backpropagation process.
[0030] In some embodiments, the short text dataset includes long text samples, second cue word samples, and second token cue samples. The step of performing a second-stage training on the first text compression model using the long text dataset to obtain a second text compression model includes:
[0031] The long text sample, the second prompt word sample, and the second token prompt sample are input into the first text compression model for forward propagation to obtain the token identifier of the second text to be predicted.
[0032] Based on the token identifier of the second text to be predicted and the second token prompt sample, a second cross-entropy loss function is constructed;
[0033] Backpropagation is performed in the first text compression model using the second cross-entropy loss function. During the backpropagation process, the parameters of the text compression unit model in the first text compression model are frozen, and the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated.
[0034] This application also provides a long text generation device based on large language model context compression, the device including the following modules:
[0035] The encoding module is used to obtain the context text to be compressed and the prompt word text, and to perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compression vector. The context text to be compressed is the text used by the large language model to perform text processing tasks.
[0036] The decoding module is used to concatenate the compressed vector with the prompt word embedding vector corresponding to the prompt word text, and perform autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers;
[0037] The generation module is used to map the target token identifiers one by one into text strings according to a preset vocabulary, and to assemble the text strings into compressed context text.
[0038] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the long text generation method based on large language model context compression as described above.
[0039] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the long text generation method based on large language model context compression as described above.
[0040] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the long text generation method based on large language model context compression as described above.
[0041] This application provides a long text generation method based on context compression using a large language model. By performing compression-based encoding on the context text to be compressed, a corresponding compressed vector is obtained, thus saving text data storage and computational resources. Next, through decoding, the compressed vector and the prompt word embedding vector are decoded into token identifiers, which are then mapped back to text strings as the compressed context text. By compressing the text, the large language model can reduce the consumption of model computational and data storage resources when performing text processing tasks. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 is a flowchart illustrating the long text generation method based on context compression of a large language model provided in this application.
[0044] Figure 2 is a schematic diagram of the principle of the long text generation method based on context compression of a large language model provided in this application.
[0045] Figure 3 is a schematic diagram of the text compression process provided in this application.
[0046] Figure 4 is a schematic diagram of the compressed vector decoding process provided in this application.
[0047] Figure 5 is a schematic diagram of the model training process provided in this application.
[0048] Figure 6 is a schematic diagram of the long text generation device based on large language model context compression provided in this application.
[0049] Figure 7 is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0051] The following describes the long text generation method and apparatus based on large language model context compression provided in this application, with reference to Figures 1-6.
[0052] Figure 1 is a flowchart of the long text generation method based on large language model context compression provided in this application. As shown in Figure 1, the method includes the following steps 101 to 103.
[0053] Step 101: Obtain the context text to be compressed and the prompt text, and perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compressed vector.
[0054] In some embodiments, such as text-based question-answering scenarios, large language models need to perform text processing tasks based on the input context text and prompt words, such as in text-based question answering and chat. The context text to be compressed is the text used by the large language model to perform text processing tasks; this text is considered long text, for example, text exceeding a certain character threshold is called long text. The prompt words are the text that helps the large language model perform text processing tasks.
[0055] Next, the context text to be compressed will be encoded using compression to obtain the corresponding compressed vector, which will be explained in detail below.
[0056] First, each character of the context text to be compressed is converted into a token identifier, resulting in a corresponding token list. The conversion process can be based on a predefined vocabulary. The token identifiers are also called token IDs, and the token IDs are combined to form the token list. Since the context text to be compressed is long, the token list is segmented according to a predefined sequence length, and the resulting subsequences are stored sequentially in a queue. The predefined sequence length can be N. If the sequence length is less than N during the final segmentation, it is padded with tokens using a compression vector to ensure that each subsequence has a length of N. During segmentation, the resulting subsequences are stored sequentially in the queue, and then a subsequence is selected from the queue and input into the encoder network for encoding. The encoded vectors are then concatenated to obtain a concatenated vector. Finally, the concatenated vector is input into a multilayer perceptron for feature dimension mapping to obtain the corresponding compressed vector. Here, the encoder network can be a Transformer model or a Mamba model. Storing and selecting subsequences in a queue prevents the order of the subsequences in the context text from being disrupted, thus avoiding interference with the semantic relationships between the text. Feature dimension mapping is performed to ensure that the feature dimension of the compressed vector is consistent with the input feature dimension of the subsequent encoding model, which facilitates subsequent decoding.
[0057] For example, see Figure 2, which is a schematic diagram of the principle of the long text generation method based on large language model context compression provided in this application. In the input layer, the acquired long context text (N*S tokens) is divided into subsequences and tokens are embedded. The subsequences are then fed into the text compression layer. In the text compression layer, the subsequences are successively fed into the encoder network through multiple hidden layers for encoding processing. The vectors obtained from the encoding processing are then compressed through the hidden layers. Finally, the compressed vectors are concatenated and output.
[0058] The text compression layer's processing is shown in Figure 3. First, the long context text is converted into a token ID sequence (i.e., a token list) using a vocabulary. Then, the token ID sequence is divided into segments of length N. Sequences shorter than N are padded with padding characters (i.e., pre-padded tokens in the compressed vector). The resulting segments are then stored in a queue in positional order. Next, it is checked whether there are any elements in the queue. If so, a token sequence is read from the queue using a first-in, first-out (FIFO) method and input into the encoder network. The current token sequence's compressed vector is obtained through encoding. This process is repeated, checking for any remaining elements in the queue, until all token sequences stored in the queue are encoded into compressed vectors. When the queue is empty, all token sequences have been encoded. Finally, all compressed vectors are concatenated along the queue's order to obtain the final compressed vector.
[0059] Step 102: Concatenate the compressed vector with the prompt word embedding vector corresponding to the prompt word text, and perform autoregressive decoding on the concatenated fusion feature to obtain multiple target token identifiers.
[0060] When performing subsequence segmentation in step 101, it is necessary to pad the tokens by pre-filling the compressed vector. Therefore, the pre-filled tokens (S tokens) and the corresponding prompt texts (M tokens) also need to be embedded. The prompt texts are converted into corresponding prompt tokens through the vocabulary and embedded into corresponding prompt embedding vectors. The pre-filled tokens (S tokens) are also embedded into corresponding embedding vectors and incorporated into the compressed vector.
[0061] Then, the compressed vector is concatenated with the corresponding prompt word embedding vector to obtain the fused feature, which is used in the subsequent decoding process. Furthermore, during the decoding process, this embodiment employs an autoregressive decoding mechanism, re-inputting the previously decoded output into the decoding model to assist in subsequent decoding. Therefore, in each decoding iteration, this embodiment obtains the current cumulative predicted token identifier, which is the target token identifier output by the decoding model. This token identifier also requires embedding processing to obtain the cumulative embedding vector.
[0062] In each decoding process, the concatenated fused features are concatenated with the cumulative embedding vector corresponding to the cumulative predicted token identifier. The concatenated features are then input into the decoding model for autoregressive decoding to obtain the corresponding target token identifier. Here, the decoding model can also be a Transformer model. It should be noted that in the first decoding iteration, the cumulative predicted token identifier does not exist; therefore, the concatenated fused features are directly input into the decoding model for autoregressive decoding.
[0063] After autoregressive decoding, the decoding model outputs corresponding target tokens, which are used to generate the compressed text. According to the autoregressive decoding mechanism, each target token needs to be evaluated. If the current target token is a terminator, the decoding process is complete, and autoregressive decoding stops. If the current target token is not a terminator, the decoding process is still ongoing, and the current target token is added to the cumulative prediction token list for use in subsequent decoding processes, continuing to generate target tokens until the current target token is a terminator.
[0064] Step 103: Based on the preset vocabulary, map the target token identifiers one by one to text strings, and combine the text strings into compressed context text.
[0065] After the decoding model outputs all the target token identifiers in step 102 (i.e., after outputting the terminator), it maps each target token identifier to a text string according to the preset vocabulary. In other words, by querying the vocabulary, the token identifier is restored to the corresponding text string. Finally, the text strings are combined to form compressed context text. This compressed context text is the final result of the context text to be compressed, which can be directly used by large language models to perform text processing tasks.
[0066] As shown in Figure 2, the output of the text compression layer enters the compression information mapping layer. After feature dimension mapping by the multilayer perceptron of the compression vector mapping unit (linear mapping), the final compressed vector is output. This compressed vector then enters the compression information decoding layer. Combined with the pre-filled tokens (Stokens) and corresponding prompt text (Mtokens) of the compressed vector embedded in the token, it is input through a hidden layer to the decoding model (i.e., the Transformer language model) for decoding, outputting the corresponding target token ID. In the final output layer, based on a pre-defined vocabulary, the target token ID is mapped to the vocabulary to generate the corresponding target text, which serves as the compression context text.
[0067] The specific decoding process is shown in Figure 4. The list of prompt word token IDs and the list of cumulatively predicted token IDs are embedded into embedding vectors a and b, respectively. These are then concatenated with the text compression vector along the sequence direction to obtain the corresponding concatenation features. These features are input into the Transformer language model for decoding to obtain token IDs (i.e., target token identifiers) used to generate new words. It is then determined whether the generated token ID is a terminator. If not, this token ID is added to the list of cumulatively predicted token IDs for subsequent decoding processes, i.e., the next decoding process. If the generated token ID is determined to be a terminator, the decoding process is complete, autoregressive decoding stops, and all newly generated token IDs are mapped to text characters through a vocabulary and output, thus obtaining the target text, i.e., the compressed context text.
[0068] In this embodiment, the context text to be compressed is encoded using compression to obtain a corresponding compressed vector, thus saving text data storage and computational resources. Next, through decoding, the compressed vector and the prompt word embedding vector are decoded into token identifiers, which are then mapped back to text strings as the compressed context text. By compressing the text, large language models can reduce the consumption of model computational and data storage resources when performing text processing tasks.
[0069] Of course, the embodiments described above combine an encoder network, a multilayer perceptron, and a decoding model to jointly complete the task of compressing long texts, solving the technical problem that large language models need to consume huge model computing and data storage resources when processing long contextual texts. However, considering that the encoder network, multilayer perceptron, and decoding model process the contextual text to be compressed, and that these network models themselves also need to be trained, directly using the long contextual text to be compressed as training data would greatly increase the burden and difficulty of model training, resulting in an unsatisfactory compression result that fails to meet the basic requirements of long text compression.
[0070] Based on the above problems, this application further proposes a model training method. First, the encoder network, multilayer perceptron, and decoding model are respectively used as three processing modules: a text compression unit model, a compression vector mapping model, and a compression vector decoding model. Then, these three processing modules are integrated into a single text compression model. This text compression model is used to compress the context text to be compressed, obtaining the compressed context text obtained in step 103. This application employs a two-stage training strategy to train the text compression model, which is described in detail below.
[0071] First, a contextual text training dataset is obtained, which includes text sequences of varying lengths. Considering that the text compression unit model directly processes long texts to be compressed, it needs to possess excellent text compression capabilities, making its parameters the most difficult to train. Therefore, in the two-stage training strategy, the datasets used in the two stages are different. The first stage involves training the text compression unit model's text compression capabilities using a dataset of short texts.
[0072] Therefore, after obtaining the context text training dataset, a text length threshold can be preset (e.g., 5000 strings). Then, based on the preset text length threshold, the context text training dataset can be divided into a short text dataset and a long text dataset. Specifically, the short text dataset includes text samples with a string length of no more than 5000 strings, while the long text dataset includes text samples with a string length of more than 5000 strings.
[0073] The text compression model is trained in the first stage using a short text dataset to obtain the first text compression model. In the first stage of training, the parameters of the text compression unit model, the compression vector mapping model, and the compression vector decoding model in the first text compression model are updated. The training process of the first stage is described in detail below.
[0074] First, the short text dataset includes short text samples, first prompt word samples, and first token prompt samples. The first token prompt sample is the token obtained by compressing and decoding the short text sample; it serves as the corresponding real label for the short text sample.
[0075] During the forward propagation of training, short text samples are input into the text compression unit model of the text compression model for encoding to obtain the corresponding first compressed vector. However, before encoding, these vectors must be converted into corresponding token identifiers. Then, the first compressed vector is input into the compression vector mapping model of the text compression model for feature mapping to obtain the text compressed vector. Simultaneously, the first embedding vector of the first prompt word sample and the second embedding vector of the first token prompt sample are determined. Here, the first prompt word sample is first converted into its corresponding token identifier before embedding, while the first token prompt sample is directly embedded.
[0076] Next, the text compression vector, the first embedding vector, and the second embedding vector are concatenated along the sequence direction to obtain the first concatenated feature. The first concatenated feature is then input into the compression vector decoding model of the text compression model for decoding processing to obtain the token identifier of the first text to be predicted, which serves as the prediction result output by the text compression model after forward propagation.
[0077] Then, based on the token identifier of the first text to be predicted and the first token prompt sample, a first cross-entropy loss function L_RAE is constructed. The purpose of the cross-entropy loss function here is to maximize the prediction probability of the correct string obtained by mapping the token identifier generated from the first text to be predicted.
[0078] This is to ensure the accuracy of the token identifier prediction obtained by the text compression model during decoding.
[0079] When constructing the loss function, multiple token identifiers at the end of the token identifier sequence of the first text to be predicted are removed, and then the same number of token identifiers at the beginning of the first token prompt sample are removed. The corresponding loss function is then constructed, and the specific formula is as follows:
[0080] In formula (1) above, c represents the first token prompt sample after removing the leading token identifier, h represents the compressed text vector corresponding to the token identifier of the first text to be predicted, p represents the first embedding vector of the first prompt word sample, P represents the prediction probability of obtaining the correct text string based on the token identifier of the first text to be predicted, and Θ LLMThis represents the model parameters of the compressed vector decoding model.
[0081] Finally, the first cross-entropy loss function L is used. RAE Backpropagation is performed in the text compression model, and the parameters of the text compression unit model, compression vector mapping model, and compression vector decoding model in the text compression model are updated during the backpropagation process to obtain the first text compression model.
[0082] In each iteration of training, the first cross-entropy loss function L is calculated through forward propagation. RAE The gradient of the text compression model is calculated through backpropagation, and Adam (Adaptive Moment Estimation) is used as the gradient descent algorithm to optimize the gradient and update the model parameters of the text compression model. The text compression unit model, the compression vector mapping model, and the compression vector decoding model all participate in the parameter update.
[0083] According to the two-stage training strategy, in the second stage, the first text compression model obtained in the first stage is trained on a long text dataset to obtain the second text compression model. Because the text compression unit model in the first text compression model already possesses excellent text compression capabilities through training on the short text dataset in the first stage, there is no need to retrain the text compression unit model of the first text compression model in the second stage. Therefore, the parameters of the text compression unit model in the first text compression model are frozen, and only the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated. The training process of the second stage is explained in detail below.
[0084] Similar to the short text dataset, the long text dataset specifically includes long text samples, second cue word samples, and second token cue symbol samples. At the start of training, the long text samples, second cue word samples, and second token cue symbol samples are input into the first text compression model for forward propagation to obtain the token identifier of the second text to be predicted.
[0085] The forward propagation process is as follows: Long text samples are input into the text compression unit model of the first text compression model for encoding, resulting in a corresponding second compressed vector. However, before encoding, this vector must be converted to a corresponding token identifier. The second compressed vector is then input into the compression vector mapping model of the first text compression model for feature mapping, yielding a text compressed vector. Simultaneously, the third embedding vector of the second prompt word sample and the fourth embedding vector of the second token prompt sample are determined. Here, the second prompt word sample is first converted to its corresponding token identifier before embedding, while the second token prompt sample is directly embedded. The text compressed vector, third embedding vector, and fourth embedding vector are then concatenated along the sequence direction to obtain the second concatenated feature. This second concatenated feature is then input into the compression vector decoding model of the first text compression model for decoding, yielding the token identifier of the second text to be predicted.
[0086] Similar to the first stage, a second cross-entropy loss function is constructed based on the token identifier of the second text to be predicted and the second token prompt sample. The construction process is similar to that of the first cross-entropy loss function, and will not be described in detail here.
[0087] Finally, backpropagation is performed on the first text compression model using the second cross-entropy loss function, which is used to train the predictive generation capability of the text compression model. Therefore, during backpropagation, the parameters of the text compression unit models in the first text compression model are frozen, while the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated. In other words, the parameters of the text compression unit models in the first text compression model are fixed, and only the parameters of the subsequent compression vector mapping model and compression vector decoding model are updated during training.
[0088] For example, the training process in the second stage can be seen in Figure 5. First, the training set is sampled, the long context text is segmented, and converted into a list of token IDs. Then, it is input into the compression vector unit (i.e., the text compression unit model of the first text compression model) for encoding. The result of the encoding is input into the compression vector mapping unit (i.e., the compression vector mapping model in the first text compression model) to obtain the corresponding text compression vector. The second prompt word sample is also converted into the corresponding prompt word token ID list, and then embedded to obtain the embedding vector a. The token ID list of the sentence to be predicted (i.e., the second token prompt sample) is also embedded to obtain the embedding vector b. Then, the text compression vector, embedding vector a, and embedding vector b are concatenated along the sequence direction, and the resulting concatenated features are input into the Transformer language model (i.e., the compression vector decoding model of the first text compression model) for decoding to obtain the token ID of the corresponding sentence to be predicted (i.e., the token identifier of the second text to be predicted).
[0089] The token ID of the sentence to be predicted needs to have multiple tokens at the end removed. Cross-entropy loss is then calculated based on the corresponding logit vector and the tokens of the sentence to be predicted (with the leading tokens removed). Here, the logit vector represents the compressed text vector corresponding to the decoded token ID of the sentence to be predicted. During training, the cross-entropy loss is used for backpropagation in the model, and the model's convergence is checked. If convergence is not achieved, the next round of iterative training continues based on the sampled dataset. If convergence begins, training terminates, and the model weights, i.e., the trained model parameters, are obtained.
[0090] The trained text compression model can directly compress the context text to be compressed, obtaining the compressed context text obtained in step 103. This effectively ensures the text compression effect and meets the text compression requirements when large language models perform text processing tasks.
[0091] This application employs a two-stage training strategy to train a text compression model. In the first stage, the text compression unit model is trained using short texts to ensure its text compression capability. In the second stage, when training with long texts, the text compression unit model is frozen, which significantly reduces the GPU memory usage during training and ensures the text compression model has predictive generation capabilities. Therefore, this two-stage training strategy effectively reduces the training burden and difficulty of the model while maintaining its basic compression processing capabilities.
[0092] The long text generation apparatus based on large language model context compression provided in this application is described below. The long text generation apparatus based on large language model context compression described below can be referred to in correspondence with the long text generation method based on large language model context compression described above.
[0093] As shown in Figure 6, the long text generation device based on large language model context compression provided in this application includes an encoding module 601, a decoding module 602, and a generation module 603. Specifically, the encoding module 601 is used to acquire the context text to be compressed and the prompt word text, and to perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compression vector, wherein the context text to be compressed is the text used by the large language model to perform text processing tasks; the decoding module 602 is used to concatenate the compression vector with the prompt word embedding vector corresponding to the prompt word text, and to perform autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers; the generation module 603 is used to map the target token identifiers one by one to text strings according to a preset vocabulary, and to assemble the text strings into compressed context text.
[0094] It should be noted that the beneficial effects of the long text generation device based on large language model context compression here correspond to those of the long text generation method based on large language model context compression mentioned above. Therefore, the beneficial effects of the long text generation device based on large language model context compression will not be elaborated here.
[0095] Figure 7 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 7, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. The processor 710, communication interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a long text generation method based on large language model context compression. This method includes: acquiring the context text to be compressed and the prompt word text; performing compression-based encoding processing on the context text to be compressed to obtain a corresponding compression vector, wherein the context text to be compressed is text used by the large language model to perform text processing tasks; concatenating the compression vector with the prompt word embedding vector corresponding to the prompt word text; performing autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers; mapping each target token identifier to a text string according to a preset vocabulary, and assembling the text strings into compressed context text.
[0096] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0097] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the long text generation method based on large language model context compression provided by the above methods. The method includes: obtaining context text to be compressed and prompt word text, and performing compression-based encoding processing on the context text to be compressed to obtain a corresponding compression vector, wherein the context text to be compressed is text used by the large language model to perform text processing tasks; concatenating the compression vector with the prompt word embedding vector corresponding to the prompt word text, and performing autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers; mapping the target token identifiers one by one to text strings according to a preset vocabulary, and assembling the text strings into compressed context text.
[0098] On another front, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the long text generation method based on large language model context compression provided by the above methods. The method includes: acquiring context text to be compressed and prompt word text, and performing compression-based encoding processing on the context text to be compressed to obtain a corresponding compression vector, wherein the context text to be compressed is text used by the large language model to perform text processing tasks; concatenating the compression vector with the prompt word embedding vector corresponding to the prompt word text, and performing autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers; mapping the target token identifiers one by one to text strings according to a preset vocabulary, and assembling the text strings into compressed context text.
[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for generating long text based on context compression of a large language model, the method comprising: Obtain the context text to be compressed and the prompt word text, and perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compression vector. The context text to be compressed is the text used by the large language model to perform text processing tasks. The compressed vector is concatenated with the prompt word embedding vector corresponding to the prompt word text, and the concatenated fusion feature is subjected to autoregressive decoding processing to obtain multiple corresponding target token identifiers. According to a preset vocabulary, the target token identifiers are mapped one by one to text strings, and the text strings are combined into compressed context text.
2. The long text generation method based on large language model context compression according to claim 1, wherein, The step of performing compression-based encoding on the context text to be compressed to obtain the corresponding compressed vector includes: Each text character of the context text to be compressed is converted into a token identifier to obtain the corresponding token list; The token list is divided according to a preset sequence length, and the resulting sub-sequences are stored sequentially in a queue. Subsequences are selected from the queue and input into an encoder network for compression for encoding processing. The vectors obtained from the encoding processing are then concatenated to obtain a concatenated vector. The concatenated vector is input into a multilayer perceptron for feature dimension mapping to obtain the corresponding compressed vector.
3. The long text generation method based on large language model context compression according to claim 1, wherein, The fused features obtained by splicing are then subjected to autoregressive decoding to obtain multiple corresponding target token identifiers, including: Get the current cumulative prediction token identifier; The fused feature obtained by splicing is spliced with the cumulative embedding vector corresponding to the cumulative predicted token identifier, and the spliced feature obtained by splicing is input into the decoding model for autoregressive decoding processing to obtain the corresponding target token identifier. If the current target token identifier is a terminator, stop autoregressive decoding; if the current target token identifier is not a terminator, add the current target token identifier as the cumulative predicted token identifier.
4. The long text generation method based on large language model context compression according to claim 1, wherein, The compressed context text is obtained by calling a text compression model, which includes a text compression unit model, a compression vector mapping model, and a compression vector decoding model. The training method of the text compression model includes: Obtain the context text training dataset, and divide the context text training dataset into a short text dataset and a long text dataset according to a preset text length threshold; The text compression model is trained in the first stage using the short text dataset to obtain a first text compression model. During the first stage of training, the parameters of the text compression unit model, the compression vector mapping model, and the compression vector decoding model in the first text compression model are updated. The first text compression model is trained in the second stage using the long text dataset to obtain the second text compression model. During the second stage of training, the parameters of the text compression unit model in the first text compression model are frozen, and the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated.
5. The long text generation method based on large language model context compression according to claim 4, wherein, The short text dataset includes short text samples, first prompt word samples, and first token prompt symbol samples. The first stage of training the text compression model using the short text dataset to obtain the first text compression model includes: The short text sample is input into the text compression unit model of the text compression model for encoding processing to obtain the corresponding first compression vector. The first compression vector is then input into the compression vector mapping model of the text compression model for feature mapping to obtain the text compression vector. Determine the first embedding vector of the first prompt word sample and the second embedding vector of the first token prompt sample; The text compression vector, the first embedding vector, and the second embedding vector are concatenated along the sequence direction to obtain the first concatenation feature; The first concatenated feature is input into the compression vector decoding model of the text compression model for decoding processing to obtain the token identifier of the first text to be predicted; Based on the token identifier of the first text to be predicted and the first token prompt sample, a first cross-entropy loss function is constructed; The first text compression model is obtained by backpropagating through the first cross-entropy loss function in the text compression model and updating the parameters of the text compression unit model, compression vector mapping model and compression vector decoding model in the text compression model during the backpropagation process.
6. The long text generation method based on large language model context compression according to claim 4, wherein, The short text dataset includes long text samples, second prompt word samples, and second token prompt symbol samples. The second text compression model is obtained by training the first text compression model using the long text dataset in a second stage, including: The long text sample, the second prompt word sample, and the second token prompt sample are input into the first text compression model for forward propagation to obtain the token identifier of the second text to be predicted. Based on the token identifier of the second text to be predicted and the second token prompt sample, a second cross-entropy loss function is constructed; Backpropagation is performed in the first text compression model using the second cross-entropy loss function. During the backpropagation process, the parameters of the text compression unit model in the first text compression model are frozen, and the parameters of the compression vector mapping model and the compression vector decoding model in the first text compression model are updated.
7. A long text generation device based on large language model context compression, the device comprising: The encoding module is used to obtain the context text to be compressed and the prompt word text, and to perform compression-based encoding processing on the context text to be compressed to obtain the corresponding compression vector. The context text to be compressed is the text used by the large language model to perform text processing tasks. The decoding module is used to concatenate the compressed vector with the prompt word embedding vector corresponding to the prompt word text, and perform autoregressive decoding processing on the concatenated fusion features to obtain multiple corresponding target token identifiers; The generation module is used to map the target token identifiers one by one into text strings according to a preset vocabulary, and to assemble the text strings into compressed context text.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the long text generation method based on large language model context compression as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the long text generation method based on large language model context compression as described in any one of claims 1 to 6.
10. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the long text generation method based on large language model context compression as described in any one of claims 1 to 6.