Text processing model training method and text processing method

Abstract

Embodiments of the present description provide a text processing model training method and a text processing method. The text processing model training method comprises: determining a target document and a corresponding target document processing result, and obtaining, from the target document, a plurality of pieces of text data and text coordinate information of each piece of text data in the target document; performing encoding processing on each piece of text data, obtaining a target text code of each piece of text data, performing encoding processing on the text coordinate information, and obtaining a target coordinate code of the text coordinate information; inputting the target text code and the target coordinate code into a text processing model for text processing, and obtaining a predicted text processing result corresponding to the target document; and on the basis of the target document processing result and the predicted text processing result, training the text processing model, and obtaining a target text processing model.

Description

Text processing model training methods and text processing methods

[0001] This disclosure claims priority to Chinese Patent Application No. 202411593664.3, filed with the China Patent Office on November 8, 2024, entitled “Text Processing Model Training Method and Text Processing Method”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to the field of artificial intelligence technology, and in particular to a method for training a text processing model. One or more embodiments of this specification simultaneously relate to two text processing methods, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0003] With the continuous development of artificial intelligence technology, neural network models are being applied to various scenarios to perform tasks. For example, neural network models are being applied to text processing scenarios for text processing.

[0004] Current neural network models often produce inaccurate results when processing text data due to its complexity. Therefore, improving the accuracy of text processing results from neural network models has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a text processing model training method. One or more embodiments of this specification simultaneously relate to two text processing methods, a text processing model training device, two text processing devices, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a text processing model training method is provided, comprising:

[0007] Determine the target document and the corresponding target document processing result, and obtain multiple text data and the text coordinate information of each text data in the target document from the target document;

[0008] The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information;

[0009] The target text encoding and the target coordinate encoding are input into a text processing model for text processing to obtain the predicted text processing result corresponding to the target document.

[0010] Based on the target document processing result and the predicted text processing result, the text processing model is trained to obtain the target text processing model.

[0011] According to a second aspect of the embodiments of this specification, a text processing model training apparatus is provided, comprising:

[0012] The data acquisition module is configured to determine the target document and the corresponding target document processing result, and to acquire multiple text data and the text coordinate information of each text data in the target document from the target document;

[0013] The encoding module is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information;

[0014] The text processing module is configured to encode the target text and the target coordinates, input them into the text processing model for text processing, and obtain the predicted text processing result corresponding to the target document.

[0015] The model training module is configured to train the text processing model based on the target document processing result and the predicted text processing result to obtain the target text processing model.

[0016] According to a third aspect of the embodiments of this specification, a text processing method is provided, comprising:

[0017] Determine the document to be processed, and obtain multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed;

[0018] The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information;

[0019] The target text encoding and the target coordinate encoding are input into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed.

[0020] According to a fourth aspect of the embodiments of this specification, a text processing apparatus is provided, comprising:

[0021] The data acquisition module is configured to determine the document to be processed and acquire multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed;

[0022] The encoding module is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information;

[0023] The text processing module is configured to encode the target text and the target coordinates, input them into the target text processing model for text processing, and obtain the text processing result corresponding to the document to be processed.

[0024] According to a fifth aspect of the embodiments of this specification, a text processing method is provided, applied to a cloud-side device, comprising:

[0025] The receiving end device sends a document to be processed, and obtains multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed;

[0026] The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information;

[0027] The target text encoding and the target coordinate encoding are input into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed.

[0028] The text processing result is sent to the terminal device.

[0029] According to a sixth aspect of the embodiments of this specification, a text processing apparatus is provided, applied to a cloud-side device, comprising:

[0030] The data acquisition module is configured to receive a document to be processed sent by the end-side device, and to acquire multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed.

[0031] The encoding module is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information;

[0032] The text processing module is configured to encode the target text and the target coordinates, input them into the target text processing model for text processing, and obtain the text processing result corresponding to the document to be processed.

[0033] The text processing result sending module is configured to send the text processing result to the end-side device.

[0034] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising:

[0035] Memory and processor;

[0036] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of any of the above methods.

[0037] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of any of the above-described methods.

[0038] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0039] This specification provides one or more embodiments of a text processing model training method. Considering the inaccuracy of text processing results output by neural network models, this method, during the training process of the text processing model, can obtain multiple text data and the text coordinate information of each text data in the target document. Based on this text coordinate information, the text processing model can identify the structure and arrangement of multiple text data. Furthermore, in the process of determining the text position encoding for text data encoding and the coordinate information position encoding for text coordinate information encoding, to avoid the problem that additional position encoding for coordinate information encoding may prevent the text processing model from accurately and efficiently processing the encoded text data, this method uses text coordinate information to enable the text processing model to identify the structure and arrangement of multiple text data. Based on the text position encoding corresponding to the associated text data, the coordinate information position encoding is determined, thus avoiding additional position encoding. This enables the text processing model to accurately and efficiently process encoded text data, overcoming the problem of inaccurate text processing results caused by complex text data, thereby obtaining a target text processing model that outputs accurate text processing results and improving the accuracy of the text processing results output by the text processing model. Attached Figure Description

[0040] Figure 1 is a schematic diagram illustrating the application of a text processing method provided in an embodiment of this specification;

[0041] Figure 2 is a flowchart of a text processing model training method provided in one embodiment of this specification;

[0042] Figure 3 is a flowchart illustrating the process of determining the layout token in a text processing model training method provided in one embodiment of this specification.

[0043] Figure 4 is a schematic diagram of the model processing flow in a text processing model training method provided in an embodiment of this specification;

[0044] Figure 5 is a flowchart of the processing procedure of a text processing model training method provided in one embodiment of this specification;

[0045] Figure 6 is a flowchart of a text processing method provided in one embodiment of this specification;

[0046] Figure 7 is a flowchart of a text processing method provided in one embodiment of this specification;

[0047] Figure 8 is a schematic diagram of the structure of a text processing model training device provided in one embodiment of this specification;

[0048] Figure 9 is a schematic diagram of the structure of a text processing device provided in one embodiment of this specification;

[0049] Figure 10 is a schematic diagram of the structure of a text processing device provided in one embodiment of this specification;

[0050] Figure 11 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0051] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0052] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0053] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0054] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0055] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0056] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0057] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0058] Layout: refers to the way text is arranged and laid out in a document. This layout can be represented by coordinate information.

[0059] Token: In the Large Language Model (LLM), a "token" refers to the basic unit in text, which can be a character, a word, or a smaller semantic unit.

[0060] Document understanding is a core research area in document intelligence, mainly involving basic tasks such as document information extraction, document layout analysis, document classification, and document VQA.

[0061] Position indices are numerical codes used in large language models to represent the relative positions of words or symbols in an input sequence. They are usually combined with word vectors using positional encoding methods to capture the sequence's order information. These position indices are synonymous with position IDs.

[0062] OCR (Optical Character Recognition): Optical character recognition.

[0063] VQA (Visual Question Answering): Visual question answering.

[0064] DocVQA: Visual Q&A for Documents.

[0065] Long context: Long sequences or long contexts refer to the ability of large language models (LLMs) to understand and generate longer contextual information when processing text.

[0066] SFT (Supervised Fine-Tuning) refers to supervised fine-tuning, a deep learning strategy used on pre-trained large language models.

[0067] CELoss (Cross-Entropy Loss): Cross-entropy loss function.

[0068] MSE Loss (Mean Squared Error Loss): Mean squared error loss.

[0069] LoRA (Low-Rank Adaptation): Low-rank adaptive technique is a technique used to efficiently adjust pre-trained models.

[0070] With the continuous development of artificial intelligence technology, neural network models are being applied to various scenarios to perform tasks, such as text processing. However, current neural network models, due to the complexity of text data, can lead to inaccurate text processing results.

[0071] For example, document understanding is an important area of ​​current industrial and academic research, especially given the need for efficient processing and understanding of complex documents. In recent years, Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have made significant progress in this field. In tasks involving rich text content, methods combining LLMs with text and layout information have shown competitiveness and can outperform MLLMs for tasks such as Document-Oriented Visual Question Answering (DocVQA) and Rich Visual Document Information Extraction (VIE).

[0072] However, current large language models may produce inaccurate results due to the complexity of text data during text processing. For example, large language models may only design text pre-training tasks (next token prediction tasks) and ignore the prediction learning of the layout between the text and the model, resulting in inaccurate predictions.

[0073] To address this issue, this specification provides three solutions: the first is to treat the layout as a modality; the second is to treat the layout as a token; and the third is a multimodal large model solution.

[0074] The first approach treats layout information as an additional modality, modeling it together with the text.

[0075] The second approach represents layout information as text tags, forming an input sequence containing both text and layout information. In document understanding scenarios (including information extraction, table comprehension, QA), no image input is required; the necessary key information can be obtained solely through plain text and the corresponding layout. Furthermore, the approach is natural: input information is provided, and no rules are needed; the rest of the document understanding is handled by the engine.

[0076] The third approach involves using a multimodal large model to directly understand the document images.

[0077] However, both solutions have drawbacks:

[0078] The first approach requires modifications to the LLM architecture, which is rather cumbersome.

[0079] The second approach, by directly inputting text and layout information as tokens into the LLM, provides a more natural and commonly used way of understanding documents. However, this approach has drawbacks: with a limited length of position ID, introducing layout information requires additional allocation of position indexes (position IDs), which limits the learning ratio of text tokens. At the same time, significantly increasing the number of tokens leads to increased computation and causes a decrease in performance in long contexts, resulting in additional long text issues.

[0080] The disadvantage of the third approach is that it is difficult to guarantee the effect in document scenarios with a lot of text or in multimodal large models that do not support it. If adaptation is required, further pre-training and SFT are needed, which is costly and slow in iteration.

[0081] Based on this, this specification provides a text processing model training method. One or more embodiments of this specification involve two text processing methods, a text processing model training device, two text processing devices, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0082] Referring to Figure 1, which illustrates an application diagram of a text processing method according to an embodiment of this specification, the client 102 sends a question (the question to be processed) and the corresponding paper document (the document to be processed) to the server 104. After receiving the question and the paper document, the server 104 can obtain multiple text data and the text coordinate information (layout information) of the text data in the paper document; then, it encodes the question and text data to obtain the corresponding text token (text data encoding), and encodes the layout information to obtain the layout token (coordinate information encoding); then, it constructs a token sequence (data encoding sequence) from the multiple text tokens, and... Each text token in the token sequence is assigned a corresponding position index (text position encoding); for the layout token, the associated text data corresponding to the layout token can be determined from multiple text data, and the position encoding (coordinate information position encoding) corresponding to the layout token is determined according to the position index corresponding to the text token of the associated text data; finally, the text token, layout token and position encoding are input into the large language model for answer reasoning, so that the large language model can fully and accurately understand the text data of the paper based on the layout information and output an accurate answer (text processing result); after obtaining the answer, the server 104 sends the answer to the client 102.

[0083] Referring to Figure 2, Figure 2 shows a flowchart of a text processing model training method according to an embodiment of this specification, which specifically includes the following steps.

[0084] Step 202: Determine the target document and the corresponding target document processing result, and obtain multiple text data and the text coordinate information of each text data in the target document from the target document.

[0085] The documents mentioned in one or more embodiments of this specification include, but are not limited to, papers, novels, manuals, articles, etc.; the target document can be understood as a document used to train the text processing model.

[0086] The document processing result can be understood as the result of processing a document. The document processing result will differ depending on the application of the text processing model training method provided in this specification to different scenarios. For example, in a document information extraction scenario, the document processing result can be information extracted from the document or information summarized from the document. In a content creation scenario, the document processing result can be content created based on the document, including but not limited to a sequel to a novel or article. The target document processing result can be understood as the document processing result of the text processing model processing a target document. In one or more embodiments of this specification, the target document processing result can be the target answer corresponding to the target question, and the target document is a document associated with the target question.

[0087] Text data can be understood as the text in a document (the target document in this example), such as the text content in a paper or the text content in a manual; text coordinate information can be understood as the two-dimensional coordinate information of text data in the target document.

[0088] In one or more embodiments provided in this specification, in order to improve the applicability of this method, this method can also identify the text data contained in the target document image block and the text coordinate information of the text data, thereby processing the text data in the target document image. Specifically, the target document is a target document image.

[0089] The step of obtaining multiple text data and the text coordinate information of each text data in the target document includes:

[0090] The target document image is segmented to obtain multiple document image blocks;

[0091] A text data region containing text is marked using a text data frame, and text recognition is performed on the text data region marked by the text data frame to obtain the text data, wherein the target document image block is any one of the plurality of document image blocks;

[0092] The coordinate information of the text data frame in the target document image is used as the text coordinate information of the text data.

[0093] The target document image can be understood as the image or scanned image corresponding to the target document. A text data frame can be used to define a data frame that contains text data or a text data region within the target document image block; this text data frame can be a bounding box. The text data region can be understood as the area where the text data is located within the target document image block.

[0094] Specifically, this method can use text extraction technology to segment the target document image to obtain multiple document image blocks, and for each document image block, use a text data frame to perform text recognition, thereby obtaining the text data contained in the target document image block and the text data frame corresponding to the text data; this text extraction technology can be OCR.

[0095] Since the text data frame has already defined the coordinates of the text data in the target document, the coordinates of the text data in the target document can be used as the text coordinates of the text data.

[0096] Taking the application of the text processing model training method provided in this specification in the scenario of improving the understanding of large language models documents based on layout information tokenization as an example, the text processing model training method is explained; wherein, the target document is a paper, and the text coordinate information is layout information.

[0097] Based on this, this method uses OCR technology to process a given paper document image. OCR technology can divide the paper document image into multiple OCR blocks (i.e., document image blocks). Each OCR block can be an image block corresponding to a line or paragraph of text in the paper; that is, each OCR block can contain a line or paragraph of text data from the paper document. Alternatively, an OCR block can correspond to a cell in a table within the document; that is, an OCR block contains the text data of one cell.

[0098] For each OCR block, a text data frame can be used to define the image region corresponding to the text data (i.e., the text data region), and text recognition can be performed on the image region to identify the text data contained in the OCR block. Furthermore, since OCR technology defines the text data using bounding boxes during the text recognition process, the coordinate information of the bounding box in the document can be used as the layout information (text coordinate information) of the text data recognized in the OCR block.

[0099] In one or more embodiments provided in this specification, the target document is a document associated with a target question, the target document processing result is the target answer corresponding to the target question, and the predicted text processing result is the predicted answer corresponding to the target question;

[0100] The determination of the target document and the corresponding target document processing result includes:

[0101] Identify the target document associated with the target question, and determine the target answer corresponding to the target question;

[0102] The target question can be a question specific to the target document; for example, summarizing the target document, extracting key information from the target document, or identifying the outline of the target document. Alternatively, the target question can be a question that requires answer reasoning based on the target document; for example, the target question could be "Briefly explain what Chinese history is," and the target document could be papers, books, etc., related to Chinese history. In other words, the target document can be the reference text for the target question; the language generation model uses the target document to reason about the answer to the target question.

[0103] The target answer can be understood as the text of the answer to the target question; the target answer will also be different depending on the target question.

[0104] Specifically, the text processing model training method provided in this specification can determine the target question and the target document associated with the target question, and determine the target answer corresponding to the target question. In this way, the target question, target document and target answer are used to train the language generation model, thereby obtaining a language generation model that can accurately perform answer reasoning and meet the needs of practical application scenarios.

[0105] Step 204: Encode each text data to obtain the target text code of each text data, and encode the text coordinate information to obtain the target coordinate code of the text coordinate information.

[0106] The target text encoding can be understood as the encoding representing each text data and / or the encoding representing the position information of each text data in the input data. The target text encoding can be a vector or a matrix. The target coordinate encoding can be understood as the encoding representing the coordinate information of the text and / or the encoding representing the position information of the text coordinate information in the input data. The target coordinate encoding can be a vector or a matrix. The input data refers to the data input into the text processing model. The text processing model can perform text processing on the input data to obtain the predicted text processing result.

[0107] In one or more embodiments provided in this specification, the step of encoding the text data to obtain the target text encoding of the text data includes:

[0108] The text data is encoded to obtain the text data encoding of each text data.

[0109] A data encoding sequence is constructed based on multiple text data codes, and a text position code is determined for each text data code in the data encoding sequence, wherein the text position code indicates the position of each text data code in the data encoding sequence;

[0110] The data encoding sequence and the text position encoding are used as the target text encoding for each text data.

[0111] Among them, text data encoding can be understood as the encoding corresponding to text data. For example, the text data encoding can be a text data token, text data vector, or text data matrix corresponding to text data; coordinate information encoding can be understood as the encoding corresponding to text coordinate information. For example, the coordinate information encoding can be a coordinate information token, coordinate information vector, or coordinate information matrix corresponding to text coordinate information.

[0112] Specifically, for each piece of text data, a corresponding text tokenizer can be determined, and this tokenizer is used to segment each piece of text data into words, determining the encoding of each word, thereby achieving the encoding processing of the text data. The text tokenizer is used to determine the text data encoding of each piece of text data.

[0113] In one or more embodiments provided in this specification, in order to efficiently encode text coordinate information, this method uses model training embedding vectors to encode the text coordinate information, thereby obtaining coordinate information encoding; specifically, the encoding process of the text coordinate information to obtain the coordinate information encoding of the text coordinate information includes:

[0114] Using the coordinate encoding unit corresponding to the text processing model, the model training embedding vector corresponding to the text coordinate information is determined, and the text coordinate information is encoded based on the model training embedding vector to obtain the coordinate information encoding of the text coordinate information.

[0115] The coordinate encoding unit is used to encode text coordinate information to obtain the coordinate information encoding unit. For example, the coordinate encoding unit can be a coordinate information tokenizer. When the text coordinate information is layout information, the coordinate encoding unit can be a layout tokenizer.

[0116] Model training embedding vectors can be understood as vectors used to encode text coordinate information. For example, a model training embedding vector can be a learnable embedding (i.e., a one-single embedding). This learnable embedding is randomly initialized. Its function is to map the four-dimensional coordinates of the bounding box (i.e., text coordinate information) to a higher-dimensional token; that is, to transform the four-dimensional coordinates into a higher-dimensional format token. This model training embedding vector is a learnable embedding vector. Learnable embedding vectors are embedding vectors that are updated as the model trains; for example, embeddings dynamically adjusted during model training are called learnable embeddings. These vectors are usually assigned random values ​​or initial values ​​of a certain distribution during model initialization, and then gradually adjusted during training according to the backpropagation algorithm.

[0117] Following the previous example, this method utilizes the layout tokenizer corresponding to the large language model (i.e., the text processing model) to determine the learnable embeddings that map the four-dimensional coordinates and encode them, thereby mapping the four-dimensional coordinates into a high-dimensional layout token (i.e., coordinate information encoding).

[0118] In one or more embodiments provided in this specification, the step of encoding the text coordinate information based on the model-trained embedding vector to obtain the coordinate information encoding of the text coordinate information includes:

[0119] The text coordinate information is processed using the linear layer in the coordinate encoding unit to obtain the key vector and value vector corresponding to the text coordinate information, wherein the information processing includes projection processing;

[0120] The model training embedding vector is determined as the query vector corresponding to the text coordinate information;

[0121] The attention layer in the coordinate encoding unit is used to perform attention processing on the key vector, the value vector, and the query vector to obtain the coordinate information encoding of the text coordinate information.

[0122] Following the previous example, Figure 3 is a flowchart illustrating the process of determining the layout token in a text processing model training method provided in one embodiment of this specification. As shown in Figure 3, this method can map the layout information in each OCR block to a one layout token. The specific implementation is as follows:

[0123] 1. In the layout word segmenter, determine the bounding box corresponding to the text data in the OCR image block, and use the coordinate information of the bounding box as the layout information of the text data.

[0124] 2. Input the layout information into two linear layers (i.e., layout projectors) for projection processing to generate key and value.

[0125] 3. A learnable one-single embedding (i.e., learnable embeddings) is used as the query. The learnable embedding is randomly initialized. The purpose of the learnable embedding is to map the four-dimensional coordinates (i.e., text coordinate information) of the bounding box to a high-dimensional token. In other words, it transforms the four-dimensional coordinates into a higher-dimensional layout token.

[0126] 4. Input the key, value, and query into MLP and ATTN for attention calculation to obtain the tokens corresponding to the layout (b0, b1, and bi in Figure 3).

[0127] Based on the above steps, this method proposes a lightweight and efficient model (i.e., a layout segmenter) that maps the layout information in each OCR block to a token, reducing the token length and significantly improving the inference speed.

[0128] In one or more embodiments provided in this specification, encoding the text coordinate information to obtain the target coordinate encoding of the text coordinate information includes:

[0129] The text coordinate information is encoded to obtain the coordinate information encoding of the text coordinate information;

[0130] From the plurality of text data, determine the associated text data that is associated with the text coordinate information, and determine the coordinate information position code of the coordinate information encoding according to the text position code corresponding to the associated text data;

[0131] The coordinate information encoding and the coordinate information position encoding are used as the target coordinate encoding of the text coordinate information.

[0132] For text coordinate information, a corresponding coordinate encoding unit can be determined for each text coordinate information, and the text coordinate information can be encoded through the coordinate encoding unit to convert it into coordinate information encoding. For example, the text coordinate information can be mapped into coordinate information encoding through the coordinate encoding unit. The coordinate encoding unit is used to encode the text coordinate information to obtain the coordinate information encoding.

[0133] Continuing with the previous example, the coordinate encoding unit can be a layout tokenizer. Based on this, this method can convert the text data in each OCR block into a token using the tokenizer. At the same time, the layout information in each OCR block is mapped to a token using the layout tokenizer, thereby significantly improving the inference speed of the large language model.

[0134] The text position encoding can be understood as the position encoding of each text data in the data encoding sequence, and the text position encoding can refer to position indices.

[0135] Continuing with the previous example, the tokens corresponding to multiple text data are sorted to obtain a token sequence (i.e., the input sequence of the language generation model), and a corresponding position index (i.e., text position encoding) is assigned to each token in the sequence. This position index can be represented in the following format: P k =P q = [0,1,...,l-2,l-1].

[0136] Among them, P k and P q The position index is represented by the attention layer for assigning key and value vectors; l represents the total number of tokens for text and layout information in an OCR segment (i.e., OCR block) (l = T + L), where T represents the number of text tokens and L represents the number of layout tokens.

[0137] Among them, the coordinate information position encoding can be understood as the position encoding of each coordinate information encoding in the data encoding sequence. This coordinate information position encoding can refer to the position indices of the text coordinate information.

[0138] Specifically, in the process of determining the coordinate information position code of the coordinate information code, this method can determine the associated text data related to the text coordinate information from multiple text data, and determine the text position code corresponding to the text data code of the associated text data. The text position code can be at least one.

[0139] From at least one text location code, a target text location code is selected as the coordinate information location code. This target text location code can be any one of the at least one text location codes, the first text location code in the at least one text location code, or the last text location code in the at least one text location code. This coordinate information location code can represent the association between this coordinate information code and the text data code corresponding to the associated text data.

[0140] In one or more embodiments provided in this specification, determining the coordinate information position code of the coordinate information code based on the text position code corresponding to the associated text data includes:

[0141] Determine the character data contained in the associated text data, wherein the character data is arranged in semantic order;

[0142] From the character data arranged in semantic order, determine the target character data located at the first semantic position in the semantic order;

[0143] From the text position codes corresponding to the associated text data, determine the target text position code corresponding to the target character data, and determine the target text position code as the coordinate information position code of the coordinate information code.

[0144] The character data is at least one, and the semantic order can be understood as the combination order of words or characters in a language.

[0145] Following the previous example, considering that introducing layout tokens would bring additional burden to the position index, this method maps the layout information in each OCR block to one layout token (i.e., coordinate information encoding). For each OCR block, it determines the text data (i.e., associated text data) contained in each OCR block, and determines the multiple text tokens (i.e., text data encoding) corresponding to the text data, as well as the layout token (i.e., coordinate information encoding) corresponding to the text data.

[0146] Then, the position indices of the first text token among multiple text tokens are assigned to the corresponding layout token, eliminating the need to allocate additional position indices to the layout information. This overcomes the problem of position index burden.

[0147] The location index of this layout information is shown below: P k =P q= [0,1,...,l-2,0].

[0148] Among them, P k and P q The position index is assigned to the key and value vectors by the attention layer; l represents the total number of tokens for text and layout information in an OCR segment (l = T + L), where T represents the number of text tokens, L represents the number of layout tokens, and the last token is used as the layout token, and its position index is the same as the first index of the text token.

[0149] Step 206: Input the target text encoding and the target coordinate encoding into the text processing model for text processing to obtain the predicted text processing result corresponding to the target document.

[0150] Following the previous example, Figure 4 is a schematic diagram of the model processing flow in a text processing model training method provided in an embodiment of this specification; wherein, in Figure 4, t represents the text token, b represents the layout token, box represents the bounding box, and x and y represent coordinate information.

[0151] Based on this, it can be seen that after this method uses a layout segmenter to convert the coordinates (i.e., the boxes in Figure 4, which are layout information) corresponding to each OCR block into layout tokens (b0 and b1 in Figure 4), the layout tokens and text tokens (t0 to t4 in Figure 4) can be combined to form a data encoding sequence to obtain the input sequence; among which, <s>This is the data that indicates the start of the sequence.

[0152] The input sequence, layout token, and position index corresponding to the text token are input into a large language model for answer reasoning, thereby obtaining the predicted text processing result.

[0153] Furthermore, based on the input and output data of the large language model in Figure 4, it can be seen that the input and output data of the large language model (i.e., t0 to t4, and BOX0 and BOX1 output by the large model in Figure 4) are aligned.

[0154] In one or more embodiments provided in this specification, the step of inputting the target text encoding and the target coordinate encoding into a text processing model for text processing to obtain the predicted text processing result corresponding to the target document includes:

[0155] The target question code, the target text code, and the target coordinate code are input into a text processing model for question processing to obtain the predicted answer. The target question code is obtained by encoding the target question.

[0156] Following the previous example, we can determine the target question and the target document associated with the target question, and determine the target answer corresponding to the target question. Then, we can use the target question, target document, and target answer to train the language generation model, thereby obtaining a language generation model that can accurately perform answer reasoning and meet the needs of practical application scenarios.

[0157] This method can convert a question into a question token using a tokenizer, and then combine the question token with the text token to form a data encoding sequence. After determining the format token using a format tokenizer, the format token can be combined with the data encoding sequence to obtain the input sequence. This input sequence, along with the position indices of the question token, format token, and text token, is then input into a large language model for answer reasoning, thereby obtaining the answer to the question.

[0158] Step 208: Train the text processing model based on the target document processing result and the predicted text processing result to obtain the target text processing model.

[0159] Specifically, after obtaining the predicted text processing result, this method can calculate the loss function based on the target document processing result and the predicted text processing result, and use the loss function to adjust the model parameters of the text processing model until the model training stopping condition is reached, thus obtaining the target text processing model.

[0160] In one or more embodiments provided in this specification, in order to simultaneously enhance the token learning of both text and boxes, this method employs a Next Text and Layout token prediction to simultaneously enhance the learning of text and layout and the interaction between these two modalities, thereby training a language generation model with better performance. Specifically, the step of training the text processing model based on the target document processing result and the predicted text processing result to obtain the target text processing model includes:

[0161] The first loss function is determined using the target document processing result and the predicted text processing result;

[0162] Determine the target text coordinate information for the target document, and determine a second loss function based on the target text coordinate information and the text coordinate information;

[0163] Based on the first loss function and the second loss function, the model parameters of the text processing model are adjusted to obtain the target text processing model.

[0164] The first loss function and the second loss function can be set according to the actual scenario. For example, the first loss function can be the cross-entropy loss function, and the second loss function can be the regression loss function.

[0165] The target text coordinate information can be understood as the actual text coordinate information in the target document, and the text coordinate information used to determine the second loss function can be understood as the text coordinate information predicted by the text processing model. In one or more embodiments of this specification, the text coordinate information can also be determined from the target document processing results during the determination of the second loss function.

[0166] Continuing with the previous example, the specific training method for large language models is as follows:

[0167] 1. Determine the samples and tags. Samples: papers and questions; tags are the answers to questions and the format of papers.

[0168] 2. Samples and labels: Through the above processing, text tokens and layout tokens are obtained, and these are input into the large model for prediction;

[0169] 3. The large model makes predictions based on the input data to obtain the corresponding predictive features;

[0170] 4. The large model processes the text features in the predicted features through the LM head, thereby outputting the predicted next word or character;

[0171] In deep learning, the LM head (Language Model Head) refers to the head structure used for language model prediction. In deep learning, the LM head is a fully connected layer whose function is to predict the next word or character based on the input feature vector.

[0172] 5. The large model processes the layout features in the predicted features through the box head, thereby outputting the layout information corresponding to the next word or character. The box head is a fully connected layer whose function is to predict the layout information (i.e., text coordinate information) of the next word or character based on the input feature vector.

[0173] 6. Calculate the cross-entropy loss function and regression loss function for the predicted answer (predicted text processing result) and the layout information (i.e., text coordinate information) corresponding to the predicted answer output by the large model, and the real sample labels (i.e., real answer and real layout information), and then adjust the model parameters based on the loss function.

[0174] Specifically, the cross-entropy loss function is calculated between the predicted answer and the true answer (i.e., the result of the target document processing); the regression loss function is calculated using the layout information corresponding to the predicted answer and the true layout information (target text coordinate information) corresponding to the true answer, and the model parameters are adjusted based on the two loss functions.

[0175] It should be noted that layout token prediction is trained under supervision by converting it into coordinates Box = [x1, y1, x2, y2].

[0176] This specification provides one or more embodiments of a text processing model training method. Considering the inaccuracy of text processing results output by neural network models, this method, during the training process of the text processing model, can obtain multiple text data and the text coordinate information of each text data in the target document. Based on this text coordinate information, the text processing model can identify the structure and arrangement of multiple text data. Furthermore, in the process of determining the text position encoding for text data encoding and the coordinate information position encoding for text coordinate information encoding, to avoid the problem that additional position encoding for coordinate information encoding may prevent the text processing model from accurately and efficiently processing the encoded text data, this method uses text coordinate information to enable the text processing model to identify the structure and arrangement of multiple text data. Based on the text position encoding corresponding to the associated text data, the coordinate information position encoding is determined, thus avoiding additional position encoding. This enables the text processing model to accurately and efficiently process encoded text data, overcoming the problem of inaccurate text processing results caused by complex text data, thereby obtaining a target text processing model that outputs accurate text processing results and improving the accuracy of the text processing results output by the text processing model.

[0177] The following description, in conjunction with Figure 5, uses the application of the text processing model training method provided in this specification in a scenario of improving document understanding of large language models based on layout information tokenization as an example to further illustrate the text processing model training method. Figure 5 shows a flowchart of the processing procedure of a text processing model training method provided in one embodiment of this specification, specifically including the following steps.

[0178] Step 502: Map the layout information in each OCR block to a layout token.

[0179] The specific execution method is as follows:

[0180] 1. Determine the question provided by the user and the corresponding paper document image, and use OCR technology to divide the paper document image into multiple document image blocks (OCR blocks).

[0181] 2. Obtain the text information and layout information (coordinate information) of each document image block.

[0182] In this context, an OCR block can be an image block corresponding to a line or a paragraph of text in a paper; that is, each OCR block can contain a line or a paragraph of text data from the paper document. Alternatively, an OCR block can correspond to a cell in a table in the document; that is, an OCR block contains the text data of one cell.

[0183] For each OCR block, text recognition can be performed to identify the text data contained in that OCR block. Furthermore, since OCR technology uses bounding boxes to define the text data during text recognition, the coordinate information of the bounding box in the document can be used as the layout information (four-dimensional coordinate information) of the text data recognized in that OCR block.

[0184] 3. Use a tokenizer to convert the text data in each OCR block into a text token; and convert the question into a text token corresponding to the question.

[0185] 4. In the layout segmenter, determine the layout information corresponding to the text data in the OCR block, and input the layout information into two linear layers (i.e., layout projectors) to generate key and value.

[0186] 5. In the layout-based word segmenter, learnable one-single embeddings are used as queries.

[0187] The learnable embedding is randomly initialized. The purpose of the learnable embedding is to map the four-dimensional coordinates (i.e., text coordinate information) of the bounding box to a high-dimensional token. In other words, it transforms the four-dimensional coordinates into a higher-dimensional layout token.

[0188] 6. In the layout tokenizer, input the key, value, and query into MLP and ATTN for attention calculation to obtain the layout token corresponding to the layout.

[0189] Step 504: For each OCR block corresponding to multiple text tokens, assign the position index (position id) of the first text token among the multiple text tokens to its corresponding layout token.

[0190] The specific implementation method is as follows:

[0191] 1. Sort the tokens corresponding to multiple text data to obtain a token sequence (i.e., the input sequence of the large language model), and assign a corresponding position index to each token in the sequence.

[0192] 2. Determine the multiple text tokens and layout tokens corresponding to each OCR block.

[0193] 3. Assign the position index of the first text token among multiple text tokens to the corresponding layout token.

[0194] The position indices of the text token and layout token are shown below: P k =P q = [0,1,...,l-2,0].

[0195] Among them, P k and P q The position index is assigned to the key and value vectors by the attention layer; l represents the total number of tokens for text and layout information in an OCR segment (l = T + L), where T represents the number of text tokens, L represents the number of layout tokens, and the last token is used as the layout token, and its position index is the same as the first position index of the text token.

[0196] Step 506: Input the text token, layout token, and position index into the large language model for processing to obtain the predicted answer output by the large language model.

[0197] The specific implementation method is as follows:

[0198] 1. The large language model predicts the answer based on the input data and obtains the prediction feature mask.

[0199] The predicted feature mask includes a text mask and a bounding box mask.

[0200] 2. The text mask in the predicted feature mask is processed through the LM head to output the predicted next word or character;

[0201] The Language Model Head (LM head) refers to the head structure used for language model prediction. In deep learning, the LM head is typically a fully connected layer whose function is to predict the next word or character based on the input feature vector.

[0202] 3. Process the bounding box mask in the predicted feature mask through the box head to output the layout information corresponding to the predicted next word or character;

[0203] The box head is a fully connected layer whose function is to predict the layout information of the next word or character based on the input feature vector.

[0204] 4. Repeat steps 1-3 above until the large language model completes language reasoning and obtains the predicted answer to the question.

[0205] The predicted answer includes text data and the layout information corresponding to the text data.

[0206] Step 508: Calculate the loss function using the predicted and actual answers, and adjust the model parameters of the large language model based on the loss function to obtain the trained large language model.

[0207] The specific implementation method is as follows:

[0208] 1. Calculate the cross-entropy loss function (CE loss) using the text data in the predicted answers and the text data in the real answers (sample labels).

[0209] 2. Calculate the regression loss function (MSE loss) using the layout information in the predicted answers and the layout information in the actual answers (sample labels).

[0210] Among them, the layout token prediction is supervised training by converting it into coordinates Box = [x1, y1, x2, y2]; that is, the layout information in the real answer (sample label) is coordinates (Box = [x1, y1, x2, y2]).

[0211] 3. Based on the cross-entropy loss function and the regression loss function, the model parameters of the large language model are adjusted to obtain the trained large language model.

[0212] Based on the above steps, it can be seen that the text processing model training method in one or more embodiments of this specification provides a large language model document understanding scheme with layout information tokenization. This scheme proposes an evaluation index T-Ratio that can effectively measure the impact of introducing layout information on the learning of the original text cards, thereby accurately evaluating the impact of layout information.

[0213] The T-Ratio evaluation metric was proposed to measure the impact of introducing layout information on the original text token learning. Therefore, it is proposed to define the ratio of the actual learned text token position length (N) to the effective training length as the Text Token Ratio in the Training Tokens (T-Ratio): T-Ratio = N t / N.

[0214] In the above formula, N represents the available length of position indices. t The approximation number is determined by approximating the number of OCRs within the available length N and the average text position length T in each OCR block.

[0215] N t It is approximately equal to N_{seg}*T, where N_{seg} = N / (T+L); where L represents the position indices that are additionally allocated to layout information in each OCR segment; and T represents the average text position length in each OCR segment.

[0216] Furthermore, this method maps layout information within each OCR block to a token, significantly improving inference speed. It utilizes the text position index within the OCR block to assign a layout position index, introducing layout information without adding any additional burden of position index allocation. Simultaneously, to enhance the learning of layout tokens, a pre-training task of "predicting the next text or layout token" is introduced.

[0217] This method achieves performance parity in short document comprehension scenarios and significant advantages (10+ improvement) in long document comprehension scenarios without allocating any additional position indexes when introducing layout information. Furthermore, in terms of efficiency, inference FLOPs / MACs are significantly reduced (50%+ reduction).

[0218] Based on this, the proposed solution is a query-based layout token scheme that maps the layout information in each OCR block to a token and uses the text position index within the block to assign a layout position index. While introducing layout information, it does not add any additional position index allocation burden and can achieve 100% T-Ratio.

[0219] Furthermore, to enhance the learning of layout tokens, an autoregressive task fully adapted to large models is introduced: predicting the next text or layout token. This approach outperforms other approaches in both performance and efficiency. It achieves performance parity on Short Context document understanding datasets without allocating any additional position indices when incorporating layout information, and demonstrates significant advantages (10+ improvement) in Long Context document understanding scenarios.

[0220] Referring to Figure 6, which shows a flowchart of a text processing method according to an embodiment of this specification, specifically including the following steps.

[0221] Step 602: Determine the document to be processed, and obtain multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed.

[0222] Step 604: Encode each text data to obtain the target text code of each text data, and encode the text coordinate information to obtain the target coordinate code of the text coordinate information.

[0223] Step 606: Input the target text encoding and the target coordinate encoding into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed.

[0224] In one or more embodiments provided in this specification, the document to be processed is a document associated with the question to be processed, and the text processing result is the answer to the question to be processed;

[0225] The process of determining the document to be processed includes:

[0226] Receive a text processing request sent by the client in response to a text processing operation, wherein the text processing request carries the problem to be processed and the document to be processed associated with the problem to be processed;

[0227] The step of inputting the target text encoding and the target coordinate encoding into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed includes:

[0228] The code for the problem to be processed, the code for the target text, and the code for the target coordinates are input into the target text processing model to process the problem and obtain the answer to the problem to be processed. The code for the problem to be processed is obtained by encoding the problem to be processed.

[0229] In one or more embodiments provided in this specification, after inputting the target text encoding and the target coordinate encoding into a target text processing model for text processing to obtain the text processing result corresponding to the document to be processed, the method further includes:

[0230] The text processing result is sent to the client so that the client can display the text processing result through the text processing interface.

[0231] The text processing interface can be understood as an interface that facilitates text processing for users. This text processing interface can be a webpage or an application interface.

[0232] This specification provides a text processing method through one or more embodiments. During text processing, this method can obtain multiple text data and their coordinate information within the document to be processed. Based on this coordinate information, the text processing model can identify the structure and arrangement of the multiple text data. Furthermore, in determining the text position encoding for the text data and the coordinate information position encoding for the coordinate information, to avoid the problem that additional position encoding for the coordinate information might prevent the text processing model from accurately and efficiently processing the encoded text data, this method determines the coordinate information position encoding based on the text position encoding corresponding to the associated text data associated with the coordinate information. This avoids additional position encoding, enabling the text processing model to accurately and efficiently process the encoded text data, overcoming the problem of inaccurate text processing results caused by complex text data, and improving the accuracy of the text processing results output by the text processing model.

[0233] The above is an illustrative scheme of a text processing method according to this embodiment. It should be noted that the technical solution of this text processing method belongs to the same concept as the technical solution of the text processing model training method described above. For details not described in detail in the technical solution of the text processing method, please refer to the description of the technical solution of the text processing model training method described above.

[0234] Referring to Figure 7, which shows a flowchart of another text processing method provided according to an embodiment of this specification, the other text processing method is applied to a cloud-side device and specifically includes the following steps.

[0235] Step 702: Receive the document to be processed sent by the receiving end device, and obtain multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed;

[0236] Step 704: Encode each text data to obtain the target text code of each text data, and encode the text coordinate information to obtain the target coordinate code of the text coordinate information;

[0237] Step 706: Input the target text encoding and the target coordinate encoding into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed;

[0238] Step 708: Send the text processing result to the terminal device.

[0239] In one or more embodiments provided in this specification, the cloud-side device can be a central cloud device in a distributed architecture or an edge cloud device in a distributed architecture. The cloud-side device can be a cloud-side device with a cloud desktop system or cloud desktop software installed and deployed, such as a cloud server or cloud host. The endpoint device can be understood as any terminal that interacts with the cloud-side device. This terminal can be a laptop, desktop computer, tablet, smart device, server, etc.

[0240] This specification provides one or more embodiments of another text processing method applied to cloud-side devices. During text processing, multiple text data and the text coordinate information of each text data in the document to be processed can be obtained from the document to be processed sent by the edge device. Based on the text coordinate information, the text processing model can identify the structure and arrangement relationship of multiple text data. Furthermore, in the process of determining the text position code for text data encoding and the coordinate information position code for text coordinate information encoding, in order to avoid the problem that the text processing model may not be able to accurately and efficiently process the text data encoding due to the additional position code added for the coordinate information encoding, this method determines the coordinate information position code of the coordinate information encoding according to the text position code corresponding to the associated text data associated with the text coordinate information, thereby avoiding the additional position code added. This enables the text processing model to accurately and efficiently process the text data encoding, overcomes the problem of inaccurate text processing results caused by the complexity of text data, improves the accuracy of the text processing results output by the text processing model, and sends the accurate text processing results to the edge device.

[0241] The above is an illustrative scheme of another text processing method in this embodiment. It should be noted that the technical solution of this other text processing method belongs to the same concept as the technical solution of the text processing model training method described above. For details not described in detail in the technical solution of the other text processing method, please refer to the description of the technical solution of the text processing model training method described above.

[0242] Corresponding to the above method embodiments, this specification also provides an embodiment of a text processing model training device. Figure 8 shows a schematic diagram of the structure of a text processing model training device provided in one embodiment of this specification. As shown in Figure 8, the device includes:

[0243] The data acquisition module 802 is configured to determine the target document and the corresponding target document processing result, and to acquire multiple text data and the text coordinate information of each text data in the target document from the target document;

[0244] The encoding module 804 is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information.

[0245] The text processing module 806 is configured to encode the target text and the target coordinates, input them into the text processing model for text processing, and obtain the predicted text processing result corresponding to the target document.

[0246] The model training module 808 is configured to train the text processing model based on the target document processing result and the predicted text processing result to obtain the target text processing model.

[0247] Optionally, the encoding module 804 is configured as follows:

[0248] The text data is encoded to obtain the text data encoding of each text data.

[0249] A data encoding sequence is constructed based on multiple text data codes, and a text position code is determined for each text data code in the data encoding sequence, wherein the text position code indicates the position of each text data code in the data encoding sequence;

[0250] The data encoding sequence and the text position encoding are used as the target text encoding for each text data.

[0251] Optionally, the encoding module 804 is configured as follows:

[0252] The text coordinate information is encoded to obtain the coordinate information encoding of the text coordinate information;

[0253] From the plurality of text data, determine the associated text data that is associated with the text coordinate information, and determine the coordinate information position code of the coordinate information encoding according to the text position code corresponding to the associated text data;

[0254] The coordinate information encoding and the coordinate information position encoding are used as the target coordinate encoding of the text coordinate information.

[0255] Optionally, the encoding module 804 is configured as follows:

[0256] Using the coordinate encoding unit corresponding to the text processing model, the model training embedding vector corresponding to the text coordinate information is determined, and the text coordinate information is encoded based on the model training embedding vector to obtain the coordinate information encoding of the text coordinate information.

[0257] Optionally, the encoding module 804 is configured as follows:

[0258] The text coordinate information is processed using the linear layer in the coordinate encoding unit to obtain the key vector and value vector corresponding to the text coordinate information, wherein the information processing includes projection processing;

[0259] The model training embedding vector is determined as the query vector corresponding to the text coordinate information;

[0260] The attention layer in the coordinate encoding unit is used to perform attention processing on the key vector, the value vector, and the query vector to obtain the coordinate information encoding of the text coordinate information.

[0261] Optionally, the target document is a target document image;

[0262] Optionally, the data acquisition module 802 is configured to:

[0263] The target document image is segmented to obtain multiple document image blocks;

[0264] A text data region containing text is marked using a text data frame, and text recognition is performed on the text data region marked by the text data frame to obtain the text data, wherein the target document image block is any one of the plurality of document image blocks;

[0265] The coordinate information of the text data frame in the target document image is used as the text coordinate information of the text data.

[0266] Optionally, the target document is a document associated with the target question, the target document processing result is the target answer corresponding to the target question, and the predicted text processing result is the predicted answer corresponding to the target question;

[0267] The data acquisition module 802 is configured as follows:

[0268] Identify the target document associated with the target question, and determine the target answer corresponding to the target question;

[0269] The text processing module 806 is configured as follows:

[0270] The target question code, the target text code, and the target coordinate code are input into a text processing model for question processing to obtain the predicted answer. The target question code is obtained by encoding the target question.

[0271] Optionally, the encoding module 804 is configured as follows:

[0272] Determine the character data contained in the associated text data, wherein the character data is arranged in semantic order;

[0273] From the character data, determine the target character data located at the first semantic position in the semantic order;

[0274] From the text position codes corresponding to the associated text data, determine the target text position code corresponding to the target character data, and determine the target text position code as the coordinate information position code of the coordinate information code.

[0275] Optionally, the model training module 808 is configured as follows:

[0276] The first loss function is determined using the target document processing result and the predicted text processing result;

[0277] Determine the target text coordinate information for the target document, and determine a second loss function based on the target text coordinate information and the text coordinate information;

[0278] Based on the first loss function and the second loss function, the model parameters of the text processing model are adjusted to obtain the target text processing model.

[0279] This specification provides one or more embodiments of a text processing model training apparatus and a text processing model training method. Considering the inaccuracy of text processing results output by neural network models, this method, during the training of the text processing model, can obtain multiple text data and the text coordinate information of each text data in the target document. Based on this text coordinate information, the text processing model can recognize the structure and arrangement of multiple text data. Furthermore, in the process of determining text position encoding for text data and determining coordinate information position encoding for text coordinate information, to avoid additional overhead for coordinate information encoding... Adding positional encoding can lead to problems where text processing models cannot accurately and efficiently process encoded text data. Therefore, this method uses text coordinate information to enable text processing models to identify the structure and arrangement of multiple text data. Based on the text positional encoding corresponding to the associated text data, the method determines the coordinate information positional encoding of the coordinate information, thereby avoiding the need for additional positional encoding. This enables text processing models to accurately and efficiently process encoded text data, overcoming the problem of inaccurate text processing results caused by complex text data. As a result, the method obtains a target text processing model that outputs accurate text processing results, improving the accuracy of the text processing results output by the text processing model.

[0280] The above is an illustrative scheme of a text processing model training device according to this embodiment. It should be noted that the technical solution of this text processing model training device and the technical solution of the text processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the text processing model training device, please refer to the description of the technical solution of the text processing model training method described above.

[0281] Corresponding to the above method embodiments, this specification also provides a text processing device embodiment. Figure 9 shows a schematic diagram of the structure of a text processing device provided in one embodiment of this specification. As shown in Figure 9, the device includes:

[0282] The data acquisition module 902 is configured to determine the document to be processed and acquire multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed;

[0283] The encoding module 904 is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information.

[0284] The text processing module 906 is configured to encode the target text and the target coordinates, input them into the target text processing model for text processing, and obtain the text processing result corresponding to the document to be processed.

[0285] Optionally, the document to be processed is a document associated with the question to be processed, and the text processing result is the answer to the question to be processed;

[0286] The data acquisition module 902 is further configured to:

[0287] Receive a text processing request sent by the client in response to a text processing operation, wherein the text processing request carries the problem to be processed and the document to be processed associated with the problem to be processed;

[0288] The text processing module 906 is further configured to:

[0289] The code for the problem to be processed, the code for the target text, and the code for the target coordinates are input into the target text processing model to process the problem and obtain the answer to the problem to be processed. The code for the problem to be processed is obtained by encoding the problem to be processed.

[0290] Optionally, the text processing device further includes a text processing result sending module, configured to:

[0291] The text processing result is sent to the client so that the client can display the text processing result through the text processing interface.

[0292] This specification provides a text processing apparatus in one or more embodiments. During text processing, it can obtain multiple text data and text coordinate information of each text data in the document to be processed. Based on the text coordinate information, the text processing model can identify the structure and arrangement relationship of the multiple text data. In the process of determining the text position code for text data encoding and the coordinate information position code for text coordinate information encoding, in order to avoid the problem that the text processing model may not be able to accurately and efficiently process the text data encoding due to the additional position code added for the coordinate information encoding, this method determines the coordinate information position code of the coordinate information encoding based on the text position code corresponding to the associated text data associated with the text coordinate information. This avoids the additional position code, enables the text processing model to accurately and efficiently process the text data encoding, overcomes the problem of inaccurate text processing results caused by the complexity of text data, and improves the accuracy of the text processing results output by the text processing model.

[0293] The above is an illustrative scheme of a text processing device according to this embodiment. It should be noted that the technical solution of this text processing device and the technical solution of the text processing method described above belong to the same concept. For details not described in detail in the technical solution of the text processing device, please refer to the description of the technical solution of the text processing method described above.

[0294] Corresponding to the above method embodiments, this specification also provides another embodiment of a text processing device. Figure 10 shows a schematic diagram of the structure of another text processing device provided in one embodiment of this specification. As shown in Figure 10, this device is applied to cloud-side equipment and includes:

[0295] The data acquisition module 1002 is configured to receive a document to be processed sent by the end-side device, and to acquire multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed.

[0296] The encoding module 1004 is configured to encode each text data to obtain the target text encoding of each text data, and to encode the text coordinate information to obtain the target coordinate encoding of the text coordinate information.

[0297] The text processing module 1006 is configured to input the target text encoding and the target coordinate encoding into the target text processing model for text processing, and obtain the text processing result corresponding to the document to be processed;

[0298] The text processing result sending module 1008 is configured to send the text processing result to the end device.

[0299] This specification provides one or more embodiments of another text processing device applied to cloud-side devices. During text processing, it can obtain multiple text data and text coordinate information of each text data in the document to be processed from the document sent by the end-side device. Based on the text coordinate information, the text processing model can identify the structure and arrangement relationship of multiple text data. In the process of determining the text position code for text data encoding and the coordinate information position code for text coordinate information encoding, in order to avoid the problem that the text processing model may not be able to accurately and efficiently process the text data encoding due to the additional position code added for the coordinate information encoding, this method determines the coordinate information position code of the coordinate information encoding according to the text position code corresponding to the associated text data associated with the text coordinate information, thereby avoiding the additional position code, enabling the text processing model to accurately and efficiently process the text data encoding, overcoming the problem of inaccurate text processing results caused by the complexity of text data, improving the accuracy of the text processing results output by the text processing model, and sending the accurate text processing results to the end-side device.

[0300] The above is an illustrative scheme of another text processing device according to this embodiment. It should be noted that the technical solution of this other text processing device and the technical solution of the other text processing method described above belong to the same concept. For details not described in detail in the technical solution of the other text processing device, please refer to the description of the technical solution of the other text processing method described above.

[0301] Figure 11 shows a structural block diagram of a computing device 1100 according to an embodiment of this specification. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0302] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0303] In one embodiment of this specification, the aforementioned components of the computing device 1100, as well as other components not shown in FIG11, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG11 is merely for illustrative purposes and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0304] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1100 can also be a mobile or stationary server.

[0305] The processor 1120 is configured to execute computer-executable instructions that, when executed by the processor, implement the steps of any of the methods described above.

[0306] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0307] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0308] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to any of the above-described method embodiments, so the description is relatively simple; relevant parts can be referred to in the description of any of the above-described method embodiments.

[0309] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0310] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of any of the above methods. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of any of the above methods.

[0311] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0312] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0313] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0314] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0315] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / s>

Claims

1. A method for training a text processing model, comprising: Determine the target document and the corresponding target document processing result, and obtain multiple text data and the text coordinate information of each text data in the target document from the target document; The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information; The target text encoding and the target coordinate encoding are input into a text processing model for text processing to obtain the predicted text processing result corresponding to the target document. Based on the target document processing result and the predicted text processing result, the text processing model is trained to obtain the target text processing model.

2. The text processing model training method according to claim 1, wherein encoding the text data to obtain the target text encoding of the text data includes: The text data is encoded to obtain the text data encoding of each text data. A data encoding sequence is constructed based on multiple text data codes, and a text position code is determined for each text data code in the data encoding sequence, wherein the text position code indicates the position of each text data code in the data encoding sequence; The data encoding sequence and the text position encoding are used as the target text encoding for each text data.

3. The text processing model training method according to claim 1, wherein encoding the text coordinate information to obtain the target coordinate encoding of the text coordinate information includes: The text coordinate information is encoded to obtain the coordinate information encoding of the text coordinate information; From the plurality of text data, determine the associated text data that is associated with the text coordinate information, and determine the coordinate information position code of the coordinate information encoding according to the text position code corresponding to the associated text data; The coordinate information encoding and the coordinate information position encoding are used as the target coordinate encoding of the text coordinate information.

4. The text processing model training method according to claim 3, wherein encoding the text coordinate information to obtain the coordinate information encoding of the text coordinate information includes: Using the coordinate encoding unit corresponding to the text processing model, the model training embedding vector corresponding to the text coordinate information is determined, and the text coordinate information is encoded based on the model training embedding vector to obtain the coordinate information encoding of the text coordinate information.

5. The text processing model training method according to claim 4, wherein the step of encoding the text coordinate information based on the model-trained embedding vector to obtain the coordinate information encoding of the text coordinate information includes: The text coordinate information is processed using the linear layer in the coordinate encoding unit to obtain the key vector and value vector corresponding to the text coordinate information, wherein the information processing includes projection processing; The model training embedding vector is determined as the query vector corresponding to the text coordinate information; The attention layer in the coordinate encoding unit is used to perform attention processing on the key vector, the value vector, and the query vector to obtain the coordinate information encoding of the text coordinate information.

6. The text processing model training method according to any one of claims 1 to 5, wherein the target document is a target document image; The step of obtaining multiple text data and the text coordinate information of each text data in the target document includes: The target document image is segmented to obtain multiple document image blocks; A text data region containing text is marked using a text data frame, and text recognition is performed on the text data region marked by the text data frame to obtain the text data, wherein the target document image block is any one of the plurality of document image blocks; The coordinate information of the text data frame in the target document image is used as the text coordinate information of the text data.

7. The text processing model training method according to any one of claims 1 to 6, wherein the target document is a document associated with a target question, the target document processing result is the target answer corresponding to the target question, and the predicted text processing result is the predicted answer corresponding to the target question; The determination of the target document and the corresponding target document processing result includes: Identify the target document associated with the target question, and determine the target answer corresponding to the target question; The step of encoding the target text and the target coordinates, inputting them into a text processing model for text processing, and obtaining the predicted text processing result corresponding to the target document includes: The target question code, the target text code, and the target coordinate code are input into a text processing model for question processing to obtain the predicted answer. The target question code is obtained by encoding the target question.

8. The text processing model training method according to any one of claims 3 to 5, wherein determining the coordinate information position code of the coordinate information code based on the text position code corresponding to the associated text data includes: Determine the character data contained in the associated text data, wherein the character data is arranged in semantic order; From the character data, determine the target character data located at the first semantic position in the semantic order; From the text position codes corresponding to the associated text data, determine the target text position code corresponding to the target character data, and determine the target text position code as the coordinate information position code of the coordinate information code.

9. The text processing model training method according to any one of claims 1 to 8, wherein training the text processing model based on the target document processing result and the predicted text processing result to obtain a target text processing model includes: The first loss function is determined using the target document processing result and the predicted text processing result; Determine the target text coordinate information for the target document, and determine a second loss function based on the target text coordinate information and the text coordinate information; Based on the first loss function and the second loss function, the model parameters of the text processing model are adjusted to obtain the target text processing model.

10. The text processing model training method according to claim 9, further comprising: The first loss function is the cross-entropy loss function, and the second loss function is the mean squared error loss function. When adjusting the model parameters of the text processing model based on the first and second loss functions, a weighted summation method is used to calculate the total loss function. The sum of the weight coefficients of the cross-entropy loss function and the mean squared error loss function is 1. The weight coefficients are dynamically adjusted according to the number of training iterations of the text processing model. In the early stage of iteration, the weight coefficient of the mean squared error loss function is not lower than the weight coefficient of the cross-entropy loss function, and in the later stage of iteration, the weight coefficient of the cross-entropy loss function is not lower than the weight coefficient of the mean squared error loss function.

11. The text processing model training method according to any one of claims 1 to 3, the method further comprising: The text coordinate information is the four-dimensional bounding box coordinate information corresponding to the text data in the document. When the coordinate encoding unit corresponding to the text processing model determines the model training embedding vector corresponding to the text coordinate information, the model training embedding vector is a randomly initialized learnable embedding vector, and the learnable embedding vector is dynamically updated during the training process of the text processing model.

12. A text processing method, comprising: Determine the document to be processed, and obtain multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed; The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information; The target text encoding and the target coordinate encoding are input into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed.

13. The text processing method according to claim 12, wherein the document to be processed is a document associated with the question to be processed, and the text processing result is the answer to the question to be processed; The process of determining the document to be processed includes: Receive a text processing request sent by the client in response to a text processing operation, wherein the text processing request carries the problem to be processed and the document to be processed associated with the problem to be processed; The step of inputting the target text encoding and the target coordinate encoding into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed includes: The code for the problem to be processed, the code for the target text, and the code for the target coordinates are input into the target text processing model to process the problem and obtain the answer to the problem to be processed. The code for the problem to be processed is obtained by encoding the problem to be processed.

14. The text processing method according to claim 12 or 13, after inputting the target text encoding and the target coordinate encoding into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed, further includes: The text processing result is sent to the client so that the client can display the text processing result through the text processing interface.

15. The text processing method according to any one of claims 12 to 14, the method further comprising: When encoding each text data to obtain the target text encoding, a text segmenter consistent with the pre-training stage of the target text processing model is used. The text segmenter's word segmentation rules for text data include: dividing character data according to semantic units, and the text data encoding length corresponding to a single semantic unit does not exceed a preset word segmentation length threshold.

16. A text processing method applied to a cloud-based device, comprising: The receiving end device sends a document to be processed, and obtains multiple text data and the text coordinate information of each text data in the document to be processed from the document to be processed; The text data is encoded to obtain the target text encoding of each text data, and the text coordinate information is encoded to obtain the target coordinate encoding of the text coordinate information; The target text encoding and the target coordinate encoding are input into the target text processing model for text processing to obtain the text processing result corresponding to the document to be processed. The text processing result is sent to the terminal device.

17. The text processing method according to claim 16, further comprising, after the receiving end device sends the document to be processed: The document to be processed is format-validated. If the document to be processed is not in image format, it is converted into the target document image format. If the document to be processed is in image format, the image clarity of the document to be processed is detected. If the clarity is lower than a preset clarity threshold, the document to be processed is image enhancement processing before the steps of obtaining text data and text coordinate information are executed.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 17.

Images