Multimodal document slicing method and related apparatus

Abstract

A multimodal document slicing method, which is applied to the technical field of artificial intelligence (AI). In the multimodal document slicing method, a plurality of text segments are obtained by means of segmenting text data in a multimodal document; and other modal data (such as images or tables) in the multimodal document is processed by means of a model, such that content comprised in the other modal data can be described in the form of text. In this way, by means of calculating the correlation degree between the plurality of text segments obtained by means of segmentation and descriptive text of the other modal data, the association between the text segments and the other modal data can be realized, and thus, an association relationship is established for sliced data on the basis of slicing different modal data in the document, thereby meeting the fine-grained retrieval requirement of a user for the multimodal document.

Description

A method and related apparatus for slicing multimodal documents

[0001] This application claims priority to Chinese Patent Application No. 202411551396.9, filed with the State Intellectual Property Office of China on October 31, 2024, entitled "A Method and Apparatus for Slicing Multimodal Documents", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for slicing multimodal documents. Background Technology

[0003] With the development of the internet, people are generating more and more data in their daily lives, such as product brochures, papers, articles, news, and other types of documents. Furthermore, people frequently need to use the internet to retrieve various documents to meet their needs.

[0004] Since document data typically contains a large amount of text, current methods usually achieve association between document data based on the text within the document data. That is, if different documents contain the same or similar text, these documents are often associated together so that related documents are returned to the user when the user performs a search.

[0005] However, with the surge in document data, increasingly complex multimodal documents have emerged. Multimodal documents not only include text but also images, tables, and other modalities of data. Current document processing methods can only retrieve a limited number of relevant documents for users, failing to meet their diverse retrieval needs for multimodal documents. Summary of the Invention

[0006] This application provides a method for slicing multimodal documents, which can establish relationships between data of different modalities in the same document, thereby meeting users' fine-grained retrieval needs for multimodal documents.

[0007] Firstly, a method for slicing multimodal documents is provided for processing multimodal documents. This method includes: firstly, an execution device acquiring a multimodal document comprising data of multiple different modalities. Specifically, the multimodal document includes first modal data and second modal data, wherein the first modal data is text data and includes multiple text fragments, and the second modal data is non-text data, such as images or tables.

[0008] Then, the execution device determines the first descriptive text corresponding to the second modality data, which describes the content included in the second modality data. For example, the execution device processes the second modality data through a target model to obtain the first descriptive text, where the target model is a neural network model capable of processing images or tables, such as a visual language model.

[0009] Finally, based on the relevance of each text fragment to the first descriptive text, the execution device selects the first text fragment from the multiple text fragments to establish an association with the second modality data. For example, the execution device calculates the relevance of each text fragment to the first descriptive text, and then selects one or more text fragments with the highest relevance from the multiple text fragments to establish an association with the second modality data.

[0010] This solution segments the text data in a multimodal document to obtain multiple text fragments. Furthermore, a model is used to process other modal data (such as images or tables) within the multimodal document, enabling the description of the content of these other modal data in text form. By calculating the similarity between the segmented text fragments and the descriptive text of other modal data, associations between the text fragments and other modal data can be established. This allows for the creation of relationships between the segmented data based on different modalities within the document, thus better meeting users' fine-grained retrieval needs for multimodal documents.

[0011] In one possible implementation, the first descriptive text includes at least one of the following: a title of the second modality data, text included in the second modality data, and a summary text of the second modality data. The summary text is obtained by the first model summarizing the content included in the second modality data.

[0012] In one possible implementation, the text included in the second modality data may include key text, which is text with prominent markings within the text included in the second modality data. Specifically, key text is text included in the second modality data to indicate more critical and important content within the second modality data.

[0013] In other words, the second modality data can specifically include non-critical text and critical text. Non-critical text can be text without highlighting, while critical text is text with highlighting. Among these, highlighted text is more visually appealing to users, and therefore, highlighted text is often the more important content in the second modality data that needs to be emphasized.

[0014] In one possible implementation, during the process of the execution device selecting a first text fragment from multiple text fragments to establish an association with the second modal data, the execution device selects the first text fragment from multiple text fragments to establish an association with the second modal data based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, as well as the positions of the multiple text fragments and the second modal data in the multimodal document.

[0015] Generally, the higher the relevance of a text fragment to the first descriptive text, the higher the correlation between the text fragment and the second modality data. The smaller the positional difference between the text fragment and the second modality data (i.e., the closer the text fragment is to the second modality data), the higher the correlation between the text fragment and the second modality data tends to be.

[0016] In this scheme, by comprehensively considering the relevance of the text fragment to the first descriptive text and the position of the text fragment to the second modality data, the text fragment most relevant to the second modality data is selected to establish an association. This can select text fragments that are as close as possible to the second modality data in terms of both semantics and position, thereby selecting the text fragments most relevant to the second modality data and improving the accuracy of text fragment selection.

[0017] In one possible implementation, before establishing an association between the first text fragment and the second modal data, the execution device also determines the positions of the multiple text fragments in the multimodal document, and determines the position of the second modal data in the multimodal document.

[0018] Specifically, the execution device may determine the positions of multiple text fragments within a multimodal document during the process of segmenting the first modal data into multiple text fragments. Furthermore, the execution device may simultaneously output the positions of the second modal data within the multimodal document from the target model while processing the second modal data through a target model to obtain the first descriptive text. Of course, the execution device may also determine the positions of the multiple text fragments and the second modal data within the multimodal document using other methods.

[0019] In one possible implementation, the execution device determines at least one candidate text segment from the multiple text segments based on the positional relationship between the text segments and the second modality data. For example, the execution device determines at least one candidate text segment that is relatively close to the second modality data from the multiple text segments. That is, at least one candidate text segment can be the text segment among the multiple text segments that is closest in position to the second modality data.

[0020] Then, based on the relevance of a text fragment among at least one candidate text fragments to the first descriptive text, the execution device selects a first text fragment from the at least one candidate text fragments and establishes an association between it and the second modal data. For example, the execution device selects the first text fragment with the highest relevance from the at least one candidate text fragments and establishes an association between it and the second modal data.

[0021] In this scheme, based on the positional relationship between text fragments and second modality data, candidate text fragments that are close to the second modality data are first identified from multiple text fragments. Then, based on the correlation between the candidate text fragments and the second modality data, a first text fragment is selected to establish an association with the second modality data. This can select text fragments that are as close as possible to the second modality data both semantically and positionally, thereby selecting the text fragments most relevant to the second modality data and improving the accuracy of text fragment selection.

[0022] In one possible implementation, the execution device determines the weight of the relevance between a text segment and the first descriptive text based on the positional relationship between the text segments and the second modality data. Specifically, the closer the text segments are to the second modality data, the greater the weight of the relevance between the text segments and the first descriptive text.

[0023] Then, based on the degree of relevance between the text fragments in the multiple text segments and the first descriptive text, and the weight of the degree of relevance between the text fragments in the multiple text segments and the first descriptive text, the first text fragment is selected from the multiple text fragments to establish an association with the second modality data.

[0024] In one possible implementation, multiple text fragments include a first text fragment. To determine the similarity between the first text fragment and the first descriptive text, the execution device first performs feature extraction on the first text fragment and the first descriptive text respectively using a text feature extraction model to obtain a first feature corresponding to the first text fragment and a second feature corresponding to the first descriptive text.

[0025] Furthermore, the execution device also performs feature extraction on the first text fragment and the second modality data respectively through a multimodal data feature extraction model, obtaining the third feature corresponding to the first text fragment and the fourth feature corresponding to the first descriptive text. The input to the multimodal data feature extraction model can be data from multiple modalities such as text or images, used to extract features from the input data, thereby outputting the corresponding features.

[0026] Thus, based on the first concatenation feature and the second concatenation feature, the execution device can determine the similarity between the first text fragment and the first descriptive text, wherein the first concatenation feature is obtained based on the first feature and the third feature, and the second concatenation feature is obtained based on the second feature and the fourth feature.

[0027] In this scheme, different models are used to extract features from text fragments, second modality data, and the descriptive text of the second modality data. The features extracted by different models are then concatenated before similarity is calculated. This approach combines the processing results of multiple different models to determine the similarity, thereby improving the accuracy of the obtained similarity and ensuring the accuracy of the subsequent association established for text fragments and second modality data.

[0028] In one possible implementation, the execution device also establishes associations between text segments based on the semantics of the text segments in the multiple text segments.

[0029] In one possible implementation, the execution device extracts key information from multiple text segments using a first model. This key information is used to indicate the semantic information contained in the text segments.

[0030] Then, based on key information from multiple text fragments, the execution device determines the semantic category of each text fragment using a first model. For example, the execution device can input the key information of each text fragment into the first model, which then categorizes the multiple text fragments based on this key information. In this way, text fragments categorized into the same category by the second model have the same semantic category.

[0031] Secondly, the execution device establishes associations between text fragments that share the same semantic category among multiple text fragments.

[0032] In one possible implementation, the key information includes at least one of keywords, key sentences, and summary information of text fragments.

[0033] In one possible implementation, the execution device segments the first modality data based on text semantics to obtain multiple text fragments. That is, the segmentation of multiple text fragments is performed based on text semantics, thereby ensuring that texts with different text semantics can be separated as much as possible.

[0034] In one possible implementation, the execution device segments the first modality data using a second model to obtain multiple text fragments; wherein the first model is a large language model, the second model is a text parsing model, and the number of parameters in the third model is less than the number of parameters in the second model.

[0035] Secondly, a multimodal document slicing device is provided, comprising: an acquisition module for acquiring a multimodal document, the multimodal document including first modal data and second modal data, the first modal data being text data and including multiple text fragments, and the second modal data being non-text data; a processing module for determining a first descriptive text corresponding to the second modal data, the first descriptive text being used to describe the content included in the second modal data; and the processing module further for selecting a first text fragment from multiple text fragments and establishing an association relationship between it and the second modal data based on the correlation between the text fragments in the multiple text fragments and the first descriptive text.

[0036] In one possible implementation, the first descriptive text includes at least one of the following: a title of the second modal data, text included in the second modal data, and a summary text of the second modal data, wherein the summary text is a summary of the content included in the second modal data.

[0037] In one possible implementation, the text included in the second modality data includes text with highlighted markers.

[0038] In one possible implementation, the processing module is further configured to: select a first text fragment from multiple text fragments and establish an association between it and the second modal data based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, and the positions of the multiple text fragments and the second modal data in the multimodal document.

[0039] In one possible implementation, the processing module is further configured to: determine at least one candidate text segment from multiple text segments based on the positional relationship between the text segments in the multiple segments and the second modal data; and select a first text segment from the at least one candidate text segment and establish an association relationship with the second modal data based on the degree of relevance between the text segments in the at least one candidate text segment and the first descriptive text.

[0040] In one possible implementation, the processing module is further configured to: determine the weight of the relevance between the text segments in the multiple text segments and the first descriptive text based on the positional relationship between the text segments in the multiple text segments and the second modal data; and select the first text segment from the multiple text segments and establish an association relationship with the second modal data based on the relevance between the text segments in the multiple text segments and the first descriptive text, and the weight of the relevance between the text segments in the multiple text segments and the first descriptive text.

[0041] In one possible implementation, the multiple text fragments include a first text fragment. The processing module is further configured to: perform feature extraction on the first text fragment and the first descriptive text respectively using a text feature extraction model to obtain a first feature corresponding to the first text fragment and a second feature corresponding to the first descriptive text; perform feature extraction on the first text fragment and the second modality data respectively using a multimodal data feature extraction model to obtain a third feature corresponding to the first text fragment and a fourth feature corresponding to the first descriptive text; and determine the correlation between the first text fragment and the first descriptive text based on the first concatenation feature and the second concatenation feature, wherein the first concatenation feature is obtained based on the first feature and the third feature, and the second concatenation feature is obtained based on the second feature and the fourth feature.

[0042] In one possible implementation, the processing module is also used to: establish associations between text segments in multiple text segments based on the semantics of the text segments in multiple text segments.

[0043] In one possible implementation, the processing module is further configured to: extract key information of the text segments from multiple text segments using a first model, the key information being used to indicate the semantic information contained in the text segments; determine the semantic categories corresponding to the multiple text segments respectively using the first model based on the key information of the multiple text segments; and establish a relationship between text segments with the same semantic category among the multiple text segments.

[0044] In one possible implementation, the processing module is further configured to: segment the first modality data using a second model to obtain multiple text fragments; wherein the first model is a large language model, the second model is a text parsing model, and the number of parameters in the second model is less than the number of parameters in the first model.

[0045] Thirdly, a slicing apparatus for a multimodal document is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the slicing apparatus for the multimodal document to perform the method described above.

[0046] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0047] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0048] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0049] A seventh aspect provides a computing device comprising a multimodal document slicing device of the third aspect or a chip system of the sixth aspect, wherein the multimodal document slicing device or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0050] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0051] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0052] Figure 1 is a schematic diagram of a system architecture provided in this application;

[0053] Figure 2 is a flowchart illustrating a multimodal document slicing method provided in this application;

[0054] Figure 3 is a schematic diagram of establishing a relationship between text fragments and images provided in this application;

[0055] Figure 4 is a schematic diagram illustrating the method for determining the correlation between a text fragment and second modality data provided in this application;

[0056] Figure 5 is a schematic diagram of a method for determining the similarity between a text fragment and a first descriptive text provided in this application;

[0057] Figure 6 is a flowchart illustrating a process for segmenting long text in a multimodal document and establishing an index structure, as provided in this application.

[0058] Figure 7 is a schematic diagram of a process for extracting descriptive text from images or tables in a multimodal document according to this application;

[0059] Figure 8 is a schematic diagram of the descriptive text for generating images and tables provided in this application;

[0060] Figure 9 is a flowchart illustrating a process for establishing a relationship between text fragments and images or tables, as provided in this application.

[0061] Figure 10 is a schematic diagram of the structure of a multimodal document slicing device provided in this application;

[0062] Figure 11 is a schematic diagram of the structure of a computing device provided in this application;

[0063] Figure 12 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0064] Figure 13 is a schematic diagram of another computing device cluster provided in this application;

[0065] Figure 14 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0067] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0068] To facilitate understanding, some technical terms used in this application will be introduced below.

[0069] (1) Large Language Model

[0070] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence.

[0071] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0072] Currently, large language models are mainly composed of Transformer networks.

[0073] (2) Transformer network

[0074] Transformer networks are powerful sequence models, but the computation time and memory required increase quadratically with sequence length, significantly increasing the hardware's storage and computing power demands. Essentially, Transformer networks employ a self-attention mechanism. Self-attention is a mechanism that associates different positions within a single sequence to compute a representation of the same sequence, playing a crucial role in machine reading, abstract summarization, and image description generation.

[0075] Taking the Transformer network applied to natural language processing as an example, the Transformer network processes input data of arbitrary length and generates new feature representations of the input data, which are then converted into target words. The self-attention network layer in the Transformer network uses an attention mechanism to capture the relationships between all other words, thereby generating new feature representations for each word. The advantage of the Transformer network's self-attention network is that the attention mechanism can directly capture the relationships between all words in a sentence without considering word positions.

[0076] (3) Prompt words

[0077] Cue words originated as an input format designed by researchers for downstream tasks. Their purpose is to help pre-trained models "recall" what they "learned" during pre-training, hence the name "cue words." For large language models, cue words are the user's input, instructing the model on the task to be performed. Cue words can be a simple question, a longer text, or a set of instructions, depending on the user's specific needs. Generally, cue words are short text strings that provide context and task-related information to help the model better understand the requirements and generate the correct output. For example, in question-answering tasks, cue words might contain a description of the question or topic to help the large language model generate the correct answer. Furthermore, cue words are often designed by humans to help the large language model better understand specific tasks or domains.

[0078] In this way, when generating content, the large language model first processes the prompt words and then outputs content based on its understanding of those prompt words. The working principle of the large language model is to predict the probability of the next word appearing based on the preceding context of the user input, thereby generating the following text word by word. Therefore, the different prompt words input by the user directly affect the quality of the large language model's output. In some cases, even if the difference in the user input prompt words is only a few words, the large language model may still generate significantly different content.

[0079] (4) Visual Language Models (VLMs)

[0080] Visual language models (VLAs) are artificial intelligence models that combine computer vision and natural language processing (NLP) techniques. They aim to understand and interpret the relationships between images and text, generating accurate and vivid natural language descriptions based on images. Through deep learning, VLAs combine image and text information to build a model capable of understanding and generating the relationships between images and text.

[0081] (5) Documents

[0082] In the field of information retrieval, a document can refer to an electronic document edited by word processing software (such as a Word document or a Portable Document Format (PDF) document) or a collection of data composed of data from web pages. In general, a document can be an ordered collection of data consisting of text, images, tables, video, or audio.

[0083] (6)Embedding

[0084] In machine learning and natural language processing, embedding refers to the process of mapping high-dimensional data (such as text, images, and videos) to a low-dimensional space, resulting in an embedding vector. Simply put, an embedding vector is an N-dimensional real-valued vector that represents the input data as points in a continuous numerical space.

[0085] Taking embedding as an example—converting text into embedding vectors—the importance of embedding lies in its ability to represent the semantics of words or sentences. Embedding vectors can represent word semantics primarily because they are learned based on the patterns of word occurrence within linguistic contexts. For instance, if a word frequently appears alongside another word in certain contexts, their embedding vectors will have similar positions in the vector space, implying they share similar meanings and semantics. Furthermore, mathematical operations can be performed on embedding vectors to reveal potential relationships between objects.

[0086] (7) Embedding model

[0087] An embedding model is a model that implements the process of mapping high-dimensional data to a low-dimensional space. Generally, embedding models are usually neural network models.

[0088] With the surge in document data, increasingly complex multimodal documents have emerged. Multimodal documents not only include text but also images, tables, and other modalities of data. Current document processing methods can only retrieve a limited number of relevant documents for users, failing to meet their diverse retrieval needs for multimodal documents.

[0089] For example, in the field of science and technology, when a user performs a paper search, existing technology often uses the paper as the smallest search unit, retrieving one or more papers related to the search query. However, in many cases, a single paper may contain a lot of content, and the user may only need to retrieve certain segments of the paper relevant to their search query, without caring about the rest of the content. This would allow the user to quickly locate the relevant section within the paper. Similarly, in web search scenarios, a webpage often contains a large amount of content. When a user provides their search query, existing technology often returns one or more webpages related to that query. As a result, after clicking through to the relevant webpage, the user often needs to browse the entire page to find the content of interest.

[0090] In view of this, this application provides a method for slicing multimodal documents. Based on text semantics, it segments the text data in a multimodal document to obtain multiple semantically segmented text fragments. Furthermore, it uses a model to process other modal data (such as images or tables) within the multimodal document, enabling the description of the content of these other modal data in text form. By calculating the similarity between the segmented text fragments and the descriptive text of other modal data, it is possible to establish associations between the text fragments and other modal data. This allows for the establishment of relationships between the segmented data based on different modalities within the document, which is beneficial for meeting users' fine-grained retrieval needs for multimodal documents.

[0091] Please refer to Figure 1, which is a schematic diagram of a system architecture provided by this application. As shown in Figure 1, in this system architecture, the execution device 10 can be implemented by a single physical host (computing device) or multiple physical hosts (computing device cluster). The execution device 10 includes an accelerator 101 and a processor 102. The accelerator 101 is used to run neural network models (such as large language models or visual language models) to process the inference tasks transmitted by the processor 102. The processor 102 is used to obtain documents from the client and schedule the accelerator 101 to execute the multimodal document slicing method provided by this application on the documents, thereby completing the segmentation and association of multimodal documents.

[0092] Optionally, the execution device 10 can be used in conjunction with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0093] In addition, the system architecture also includes a data storage system 11, which is used to store data such as documents or program code.

[0094] Optionally, for persistent data storage, the data storage system 11 can be located external to the execution device 10 and exchange data with the execution device 10 via a network. Alternatively, if the execution device 10 is a physical host, the data storage system 11 can also be located internally to the execution device 10, such as by exchanging data with the processor via a bus. In this case, the data storage system 11 functions as a hard disk. With the data storage system 11, the execution device 10 can use data (such as documents) in the data storage system 11 or call program code in the data storage system to implement the multimodal document slicing method provided in this application.

[0095] Optionally, users can interact with execution device 10 using their respective local devices. For example, a client 121 is deployed on local device 12, and users interact with the execution device through client 121 on local device 12. Local device 12 can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, laptop, or smart car.

[0096] Local device 12 can interact with execution device 10 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0097] Optionally, during the process of the execution device 10 implementing the multimodal document slicing method, the local device 12 can provide documents to the execution device 10 so that the execution device 10 can process the input multimodal documents through the model to achieve multimodal document segmentation and association. Furthermore, after the execution device 10 executes the multimodal document slicing method and obtains the multimodal document slicing results, it can receive a retrieval request from the local device 12 and perform a retrieval within the multimodal document slicing results, thereby returning the retrieved document slices to the local device 12.

[0098] Please refer to Figure 2, which is a flowchart illustrating a multimodal document slicing method provided in this application. As shown in Figure 2, the multimodal document slicing method includes the following steps 201-203.

[0099] Step 201: Obtain a multimodal document. The multimodal document includes first modal data and second modal data. The first modal data is text data and includes multiple text fragments. The second modal data is non-text data.

[0100] In this application, a multimodal document can specifically be a document that includes data in multiple different modalities. Specifically, a multimodal document includes first-modal data and second-modal data with different modalities. The first-modal data is text-based, while the second-modal data is non-text-based, such as images, tables, or even audio or video. The first-modal data can be plain text data, while the second-modal data includes non-plain text data. That is, the second-modal data may include text data; for example, if the second-modal data is a table, the table may contain text data. Furthermore, the second-modal data is actually a data object, such as an image, a table, an audio file, or a video file. It is understood that a multimodal document may include combinations of multiple second-modal data objects. For example, a multimodal document can simultaneously include any combination of at least one image, at least one table, at least one audio file, or at least one video file.

[0101] In practical applications, a multimodal document can specifically be an electronic document edited using word processing software (such as a Word document or a PDF document); or, a multimodal document can also be a collection of data composed of data from web pages. For example, in the case of a multimodal document being a paper, it typically includes a large amount of text, interspersed with images and tables. In general, a multimodal document can be understood as a collection of data composed of different modalities, and this application does not limit the source of the multimodal document.

[0102] In this application, the first modal data includes all text in the multimodal document; that is, the first modal data is essentially a long text. Since different text segments within the first modal data may describe different content—for example, a paper might use different text segments to introduce various technical solutions—this application may divide the first modal data into multiple text segments.

[0103] Specifically, the execution device can segment the first modality data based on the semantics of the text, thereby obtaining multiple text fragments. In this way, each text fragment obtained from the segmentation of the first modality data can have a corresponding semantic category. It should be noted that since the text included in the first modality data is actually ordered, the segmentation based on text semantics also follows the order of the text itself within the first modality data, thus obtaining the aforementioned ordered multiple text fragments.

[0104] A text fragment may include the content of one or more chapters, one or more paragraphs, or one or more sentences. This application does not impose a specific limit on the size of the text fragment.

[0105] Step 202: Determine the first description text corresponding to the second modal data. The first description text is used to describe the content included in the second modal data.

[0106] In this application, the execution device may, for example, process second modal data through a target model to obtain first descriptive text. The first model is a neural network model capable of processing images or tables, such as a visual language model. By using the first model to process the second modal data, the first descriptive text output by the first model can be obtained. The first model can understand the content included in the second modal data (such as images or tables) and describe the content included in the second modal data in text form, thereby obtaining the first descriptive text.

[0107] For example, the first descriptive text includes at least one of the following: the title of the second modality data, the text included in the second modality data, and the summary text of the second modality data. The summary text is obtained by the first model summarizing the content included in the second modality data.

[0108] Understandably, for images or tables in a document, to facilitate user understanding of their content, images or tables often have corresponding titles, i.e., image titles or table titles. Image titles briefly summarize the content of the image and are often included within the image itself. Table titles briefly summarize the content of the table and are often included within the table itself. Therefore, by recognizing and understanding the second modality data based on the first model, the title of the second modality data (i.e., image title or table title) can be obtained. For example, in a multimodal document that is a paper about model processing, and where the second modality data is an image, the title of the second modality data could be, for example, "Figure A is a flowchart of performing XX processing on the model"; where Figure A refers to the second modality data itself, and XX processing refers to a technical solution proposed in the paper for performing processing on the model.

[0109] Furthermore, although the second modality data is essentially image-based or tabular data, in many scenarios, images often include text, and tables primarily contain text. Therefore, when the second modality data includes text, the first model, after understanding the second modality data, can extract key text indicating important content from all the text included in the second modality data.

[0110] As for the summary text, it can be understood as the content summarized in text form after the first model understands the content included in the second modality data. For example, if the second modality data is an image that indicates the model training process, the summary text could be: "The image shows a pre-training process framework diagram, and the pre-training process involves similarity matching between the input image and the text..."

[0111] Optionally, the text included in the second modality data may include key text, which is text with a prominent mark within the second modality data. Key text is text included in the second modality data to indicate the more important and crucial content within it. In other words, the second modality data can specifically include both non-key text and key text. Non-key text may be text without a prominent mark, while key text is text with a prominent mark. Text with a prominent mark is more visually appealing to users; therefore, text with a prominent mark is often the more important content in the second modality data that needs to be highlighted.

[0112] For example, the highlighting marks on key text may specifically include at least one of the following: bolding the font, using a larger font size than non-key text, highlighting the font with a bright color, using an underline, and italics. In general, the highlighting marks on key text only need to ensure that the key text is more visually noticeable than non-key text, and this application does not limit the form of the highlighting marks.

[0113] Step 203: Based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, select the first text fragment from the multiple text fragments and establish a correlation with the second modality data.

[0114] After obtaining the first descriptive text corresponding to the second modality data, since multiple text fragments and the first descriptive text are all data under the same modality (i.e., all are text), the relevance between each text fragment and the first descriptive text can be calculated separately. Then, the first text fragment can be selected from the multiple text fragments to establish an association with the second modality data. The first text fragment can include one or more text fragments from the multiple text fragments. For example, the first text fragment may include one or more text fragments with the highest relevance to the first descriptive text from the multiple text fragments. Furthermore, the relevance between the text fragment and the first descriptive text can be specifically obtained by calculating the similarity between the text fragment and the first descriptive text.

[0115] Understandably, since the first descriptive text is used to describe the content included in the second modality data, the correlation between the second modality data and the text fragment can be determined by calculating the correlation between the text fragment and the first descriptive text. The higher the correlation between the text fragment and the first description, the higher the correlation between the second modality data and the text fragment; the lower the correlation between the text fragment and the first description, the lower the correlation between the second modality data and the text fragment.

[0116] In this way, after establishing the association between the first text fragment and the second modal data, if the first text fragment is found in the search results provided by the user during the subsequent content retrieval process, the first text fragment and the second modal data associated with the first text fragment can be returned to the user simultaneously based on the established association, thereby completing the content retrieval.

[0117] Specifically, after obtaining multiple text fragments and second modal data from a multimodal document, the execution device can create corresponding indexes for each text fragment and the second modal data, enabling rapid retrieval of text fragments or second modal data based on these indexes. That is, each text fragment corresponds to a unique index, and each second modal data also corresponds to a unique index. Therefore, when establishing the association between the first text fragment and the second modal data, an association can be established between the index corresponding to each text fragment in the first text fragment and the index corresponding to the second modal data, thus realizing the association between the first text fragment and the second modal data. For example, if the first text fragment includes text fragment 1, and the index of text fragment 1 is index 1, and the index of the second modal data is index 2, the execution device can record the association between index 1 and index 2, thereby establishing the association between text fragment 1 and the second modal data.

[0118] For example, please refer to Figure 3, which is a schematic diagram of establishing a relationship between text fragments and images according to this application. As shown in Figure 3, a multimodal document includes a large amount of text (i.e., the first modality data mentioned above) and an image (i.e., the second modality data mentioned above). Based on text semantics, the large amount of text in the multimodal document can be segmented to obtain multiple text fragments, such as the ordered text fragments 1-5 in Figure 3. The image in the multimodal document is located between text fragments 2 and 3. By processing the image in the multimodal document using the first model, the first descriptive text corresponding to the image can be obtained. Thus, by calculating the similarity between each text fragment from text fragments 1 to 5 and the first descriptive text, the text fragment with the highest similarity (i.e., text fragment 2) can be selected from text fragments 1 to 5 to establish a relationship with the image.

[0119] Furthermore, the above description uses a single second modal data point as an example to illustrate how to establish a relationship between that second modal data and its associated text fragments. In practice, when a multimodal document includes a combination of multiple second modal data objects (for example, a multimodal document may simultaneously include any combination of at least one image, at least one table, at least one audio file, or at least one video file), the execution device can refer to the above processing method to establish a relationship between any second modal data point in the multimodal document and its associated text fragments.

[0120] It's important to note that in most documents, the position of an image or table is often closely related to the position of the text it's associated with. For example, when editing a document, users often describe specific content in text and then attach an image or table before or after that content. Therefore, the positional relationship between an image or table and the text fragment can actually be considered an important factor in assessing their relevance.

[0121] Optionally, when the execution device selects a first text fragment from multiple text fragments to establish an association with the second modal data, the execution device specifically selects the first text fragment from multiple text fragments to establish an association with the second modal data based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, as well as the positions of the multiple text fragments and the second modal data in the multimodal document.

[0122] For example, the execution device can first determine the positions of multiple text fragments within a multimodal document, as well as the position of the second modal data within the multimodal document. For instance, the execution device obtains positional information such as the page and paragraph number of each text fragment within the multimodal document; it also obtains information such as the page number and adjacent paragraph number of the second modal data within the multimodal document. Specifically, the execution device can determine the positions of the multiple text fragments within the multimodal document during the process of segmenting the first modal data into multiple text fragments. Furthermore, the execution device can simultaneously output the position of the second modal data within the multimodal document from the target model while processing the second modal data through the target model to obtain the first descriptive text. Of course, the execution device can also determine the positions of the multiple text fragments and the second modal data within the multimodal document using other methods.

[0123] Then, based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, as well as the positions of the multiple text fragments and the second modal data in the multimodal document, the execution device selects the first text fragment from the multiple text fragments and establishes an association between it and the second modal data.

[0124] Generally, the higher the similarity between a text fragment and the first descriptive text, the stronger the correlation between the text fragment and the second modality data. The smaller the positional difference between the text fragment and the second modality data (i.e., the closer the text fragment is to the second modality data), the stronger the correlation between them tends to be. Based on this, by comprehensively considering the relevance of a text fragment to the first descriptive text and the positional difference between the text fragment and the second modality data, one or more text fragments with the highest correlation to the second modality data can be selected as the first text fragment to be associated with.

[0125] In one possible implementation, the execution device determines at least one candidate text segment from the multiple text segments based on the positional relationship between the text segments and the second modality data. For example, the execution device determines at least one candidate text segment that is relatively close to the second modality data from the multiple text segments. That is, at least one candidate text segment can be the text segment among the multiple text segments that is closest in position to the second modality data.

[0126] Then, based on the relevance of a text fragment among at least one candidate text fragments to the first descriptive text, the execution device selects a first text fragment from the at least one candidate text fragments and establishes an association between it and the second modal data. For example, the execution device selects the first text fragment with the highest relevance from the at least one candidate text fragments and establishes an association between it and the second modal data.

[0127] In this scheme, based on the positional relationship between text fragments and second modality data, candidate text fragments that are close to the second modality data are first identified from multiple text fragments. Then, based on the correlation between the candidate text fragments and the second modality data, a first text fragment is selected to establish an association with the second modality data. This can select text fragments that are as close as possible to the second modality data both semantically and positionally, thereby selecting the text fragments most relevant to the second modality data and improving the accuracy of text fragment selection.

[0128] In another possible implementation, the execution device determines the weight of the relevance between a text segment and the first descriptive text based on the positional relationship between the text segments and the second modal data. Specifically, the closer the text segments are to the second modal data, the greater the weight of their relevance to the first descriptive text.

[0129] Then, based on the degree of relevance between the text fragments in the multiple text segments and the first descriptive text, and the weight of the degree of relevance between the text fragments in the multiple text segments and the first descriptive text, the first text fragment is selected from the multiple text fragments to establish an association with the second modality data.

[0130] For example, please refer to Figure 4, which is a schematic diagram of determining the degree of association between text fragments and second modality data provided by this application. As shown in Figure 4, for text fragments 1 to 5 obtained from the segmentation in the multimodal document, the weight corresponding to each text fragment can be determined based on the positional relationship between each text fragment and the second modality data. Furthermore, for the first descriptive text corresponding to the second modality data, the degree of correlation between each text fragment and the first descriptive text can be calculated. Finally, by multiplying the weight corresponding to each text fragment by the degree of correlation between the text fragment and the first descriptive text, the degree of association between the text fragment and the second modality data is obtained, so as to select one or more text fragments with the highest degree of association from text fragments 1 to 5 to establish an association relationship with the second modality data.

[0131] In this scheme, by combining the similarity between the text fragment and the first descriptive text, as well as the positional difference between the text fragment and the second modality data, the most relevant text fragment to the second modality data is selected to establish an association. This can select text fragments that are as close as possible to the second modality data in terms of both semantics and position, thereby selecting the most relevant text fragments to the second modality data and improving the accuracy of text fragment selection.

[0132] To make it easier to understand, the following will explain how to determine the similarity between a text fragment and the first descriptive text.

[0133] For example, please refer to Figure 5, which is a schematic diagram of determining the similarity between a text fragment and a first descriptive text according to this application. As shown in Figure 5, the aforementioned multiple text fragments include the first text fragment.

[0134] The execution device uses a text feature extraction model to extract features from the first text fragment and the first descriptive text, respectively, to obtain a first feature corresponding to the first text fragment and a second feature corresponding to the first descriptive text. The text feature extraction model, for example, is a pre-trained pure text embedding model used to extract features from the input text data, thereby obtaining text features. Generally, the closer the semantics of the first text fragment and the first descriptive text are, the higher the similarity between the first feature and the second feature extracted by the text feature extraction model for the first text fragment and the first descriptive text, respectively.

[0135] Furthermore, the execution device also performs feature extraction on the first text fragment and the second modal data using a multimodal data feature extraction model, respectively, to obtain a third feature corresponding to the first text fragment and a fourth feature corresponding to the first descriptive text. The input to the multimodal data feature extraction model can be data from multiple modalities, such as text or images, used to extract features from the input data and output corresponding features. For example, the multimodal data feature extraction model can be a pre-trained multimodal embedding model used to extract features from the input text or image to obtain output features. Generally, the closer the semantics of the first text fragment and the second modal data, the higher the similarity between the third and fourth features extracted by the multimodal data feature extraction model for the first text fragment and the second modal data, respectively.

[0136] Finally, based on the first and second concatenation features, the execution device determines the similarity between the first text fragment and the first descriptive text. The first concatenation feature is obtained based on the first and third features, and the second concatenation feature is obtained based on the second and fourth features.

[0137] In other words, this solution concatenates the features extracted by different models for the first text fragment to obtain the first concatenated feature, and concatenates the features extracted by different models for the second modality data itself or the corresponding first descriptive text to obtain the second concatenated feature. This allows the first concatenated feature to be used as the feature of the first text fragment, and the second concatenated feature to be used as the feature of the second modality data. By calculating the similarity between the first and second concatenated features, the execution device can effectively determine the degree of correlation between the first text fragment and the first descriptive text.

[0138] Optionally, in the process of obtaining the first splicing feature and the second splicing feature, the executing device may directly splice the first feature and the third feature to obtain the first splicing feature, and directly splice the second feature and the fourth feature to obtain the second splicing feature.

[0139] Alternatively, the execution device can multiply the first feature by a first weight coefficient and the third feature by a second weight coefficient before concatenating them to obtain the first concatenated feature. Similarly, the execution device can multiply the second feature by a first weight coefficient and the fourth feature by a second weight coefficient before concatenating them to obtain the second concatenated feature. The values ​​of the first and second weight coefficients can be set or adjusted according to the needs of the actual application, and this application does not impose specific limitations on them. Generally, the confidence level of a text feature extraction model is higher than that of a multimodal data feature extraction model; therefore, the first weight coefficient can be set to be greater than the second weight coefficient.

[0140] In this scheme, different models are used to extract features from text fragments, second modality data, and the descriptive text of the second modality data. The features extracted by different models are then concatenated and the relevance is calculated. This approach combines the processing results of multiple different models to determine the relevance, thereby improving the accuracy of the obtained relevance and ensuring the accuracy of the subsequent association established for text fragments and second modality data.

[0141] The above describes the process of establishing the association between text fragments and second modality data. The following will describe the process of establishing the association between text fragments.

[0142] Optionally, the execution device can also establish associations between text segments based on the semantics of the text segments in multiple text segments.

[0143] Specifically, for multiple text segments obtained from the same multimodal document, each text segment has a corresponding semantic category. That is, each text segment has a unique semantic category. The semantic category indicates the semantics of the text content included in the text segment. If two text segments contain text with different semantics, then the semantic categories corresponding to these two text segments are different.

[0144] Therefore, the execution device can establish associations between text fragments that share the same semantic category among multiple text segments. In other words, text fragments with the same semantic category will be associated with each other, thereby linking text fragments in the same multimodal document that describe the same or similar content.

[0145] When the execution device establishes associations for text fragments, the associations established between the second modal data and the text fragments, as well as the associations established between the text fragments, can be combined to associate the second modal data with more text fragments. For example, assuming the second modal data is associated with text fragment 1, and text fragment 1 is also associated with other text fragments (such as text fragment 2), then when the user retrieves any text fragment associated with text fragment 1 or text fragment 2, the system will return text fragment 1, all text fragments associated with text fragment 1, and the second modal data, thus ensuring the completeness of the search results.

[0146] Of course, in some other implementations, the association established between the second modality data and the text fragment is independent of the association established between the text fragments. That is, when a user performs a search, after retrieving a certain text fragment, the user can determine other text fragments associated with that text fragment based on the association between text fragments, or they can determine other modality data (such as images or tables) associated with that text fragment based on the association between the text fragment and other modality data.

[0147] Furthermore, the association established between the second modality data and the text fragments can also be executed based on the association established between the text fragments. Specifically, after the execution device establishes the association between the text fragments, it can group multiple text fragments with associations into the same group. Then, when establishing the association between the second modality data and the text fragments, the execution device can select representative text fragments from each group, and then select a first text fragment from the selected text fragments to establish an association with the second modality data.

[0148] Optionally, to determine the semantic category corresponding to the multiple text segments obtained from the segmentation, the execution device can extract key information from each of the multiple text segments using a first model. This key information indicates the semantic information contained in each text segment. That is, the execution device can input each of the multiple text segments into the first model, and the first model will extract the key information for each text segment.

[0149] Then, based on the key information of multiple text fragments, the execution device determines the semantic category corresponding to each text fragment through a first model. For example, the execution device can input the key information of each text fragment into the first model, which then categorizes the multiple text fragments based on the key information of each fragment. In this way, text fragments categorized into the same category by the first model have the same semantic category, while text fragments categorized into different categories have different semantic categories.

[0150] In this way, the execution device can then establish associations between text fragments that share the same semantic category among multiple text fragments.

[0151] For example, suppose a multimodal document is segmented into five text segments: text segment 1 through text segment 5. The execution device extracts key information from text segments 1 through 5 using a first model. Based on this key information, the first model categorizes the text segments: text segments 1 and 3 are assigned to the first category, text segments 2 and 5 to the second category, and text segment 4 to the third category. Therefore, the execution device can determine that text segments 1 and 3 correspond to semantic category 1, text segments 2 and 5 to semantic category 2, and text segment 4 to semantic category 3.

[0152] The key information of the text fragments extracted by the first model may include at least one of keywords, key sentences, and summary information of the text fragments.

[0153] For example, the first model may be a large language model capable of understanding text semantics. Based on pre-constructed prompts, the execution device can instruct the first model to extract key information such as keywords, key sentences, and summary information from the text fragment. For example, for any given text fragment, the prompts constructed by the execution device may specifically be: "Please understand the following text fragment and extract the corresponding keywords, key sentences, and summary information from the text fragment."

[0154] Optionally, when segmenting the first modal data, the execution device can segment the first modal data using a second model to obtain multiple text fragments. For example, the execution device inputs the first modal data into the second model, which understands the semantics of the text in the first modal data and segments the first modal data according to the text semantics, thereby outputting multiple segmented text fragments.

[0155] In this model, the first model used to extract key information from text fragments is, for example, a large language model, while the second model used to perform text fragment segmentation is a text parsing model, with fewer parameters than the first model. Specifically, the second model can be, for example, a natural language model specifically designed for text segmentation (such as a Transformer network), and its parameter count is significantly smaller than that of the large language model, thus ensuring efficient text segmentation. Furthermore, compared to using a large language model to segment the first modality data, using a dedicated second model for text segmentation can solve the illusion problem caused by excessively long input text (i.e., first modality data) in the large language model (i.e., the output of the large language model differs significantly from the user's expected result), ensuring the accuracy of text fragment segmentation.

[0156] The above describes a method for slicing multimodal documents provided in this application. To facilitate understanding, the following will detail the execution flow of this method in practical applications with specific examples.

[0157] For example, please refer to Figure 6, which is a flowchart of a process for segmenting long text in a multimodal document and building an index structure provided by this application.

[0158] As shown in Figure 6, for a multimodal document (corresponding to the multimodal document in the above embodiments), the execution device first extracts the long text (corresponding to the first modality data in the above embodiments) from the multimodal document. The execution device can extract the long text from the multimodal document in various ways, such as through a text extraction model or specialized text extraction software. The long text extracted by the execution device includes all text in the multimodal document.

[0159] Then, the execution device inputs the long text into a text parsing model (corresponding to the second model in the above embodiment). The text parsing model segments the long text based on text semantics, thereby obtaining multiple text fragments. These multiple text fragments are obtained by segmenting the long text. For each of the multiple text fragments, each text fragment can be input into a large language model (corresponding to the first model in the above embodiment). The large language model extracts key information such as keywords, key sentences, and summary information from each text fragment.

[0160] After obtaining the key information of each text segment, the key information of each text segment in multiple text segments can be input into the large language model. The large language model will then classify the multiple text segments according to the key information of each text segment, thereby grouping text segments with the same or similar semantics into the same semantic category.

[0161] Finally, after determining the semantic category to which each text segment belongs, an index structure (such as a graph-structured index) can be built for multiple text segments, thereby establishing relationships between text segments within the same semantic category. For example, when building the index structure, the execution device can create the same index for different text segments within the same semantic category, and the indices corresponding to text segments in different semantic categories will not be the same.

[0162] Furthermore, in practical applications, if finer-grained text segmentation is required, the execution device can input the segmented text fragments as longer texts into the text parsing model to execute the aforementioned process. This allows for further segmentation and the establishment of relationships between text fragments, i.e., recursive processing of text fragments. In this way, by recursively processing text fragments, finer-grained text segments can be obtained to meet the application needs of real-world scenarios.

[0163] For example, please refer to Figure 7, which is a schematic diagram of a process for extracting descriptive text from images or tables in a multimodal document provided by this application.

[0164] As shown in Figure 7, for the same multimodal document shown in Figure 6, the execution device can input the multimodal document into the visual language model, and the visual language model can extract the images and tables in the multimodal document.

[0165] For any image identified, the execution device can further process the image using a visual language model to perform fine-grained image recognition and extract image descriptive text. Specifically, the visual language model extracts information such as the image title and key text contained within the image by understanding the text in the image and the semantics of the image itself. Furthermore, the visual language model summarizes the content contained in the image based on its semantics, thus obtaining a summary text for the image. In this way, the image title, key text contained within the image, and summary text extracted by the visual language model constitute the image descriptive text.

[0166] For example, please refer to Figure 8, which is a schematic diagram of generating descriptive text for images and tables provided by this application. As shown in Figure 8, in the case where the multimodal document is a paper related to model processing, for an image in the multimodal document, the image descriptive text extracted by the visual language model is, for example, "The image shows the CILP process framework diagram. The left side is the pre-training method, and the right side is the inference process. In pre-training, the input image and text are matched for similarity...". It should be noted that the image in the paper shown in Figure 8 is only an example. Due to space limitations, the content in the image may have low clarity, but the specific content in the image does not actually affect the solution provided by this application.

[0167] For any table identified, the execution device can further process the table using a visual language model. Since the main content of the table consists of text, the visual language model can first identify and extract the text from the table, and then structure the extracted text, for example, by converting the extracted text into Structured Query Language (SQL) statements to better understand the table's content. Based on the structured text obtained after execution, the visual language model can extract the table description text, which includes, for example, the table title, key text included in the table, and a summary text of the table.

[0168] For example, as shown in Figure 8, for a table in a multimodal document, the table description text extracted by the visual language model is, for example, "The table shows a comparison of the metrics of the CLIP method and the Visual method. The CLIP method has higher metrics than the Visual method in various datasets."

[0169] After completing the indexing of text fragments and extracting the descriptive text corresponding to images or tables, the execution device can continue to establish associations between images or tables and text fragments, thereby realizing the association between different modal data.

[0170] For example, please refer to Figure 9, which is a flowchart illustrating a process for establishing associations between text fragments and images or tables provided in this application. As shown in Figure 9, by performing the processes shown in Figures 6 and 7 on a multimodal document, long text in the multimodal document can be segmented into multiple text fragments, and the descriptive text corresponding to images and tables in the multimodal document (i.e., image description text and table description text) can be extracted.

[0171] When establishing the association between an image and a text fragment, the execution device can input the text fragment and the image description text into a plain text embedding model to obtain the text features of the text fragment (corresponding to the first feature in the above embodiment) and the text features of the image description text (corresponding to the second feature in the above embodiment). Furthermore, the execution device can also input the text fragment and the image into a multimodal embedding model to obtain the multimodal features of the text fragment (corresponding to the third feature in the above embodiment) and the multimodal features of the image (corresponding to the fourth feature in the above embodiment). Thus, by concatenating the text features of the text fragment with the multimodal features of the text fragment, a first concatenated feature can be obtained; by concatenating the text features of the image description text with the multimodal features of the image, a second concatenated feature can be obtained. Based on the first and second concatenated features, the similarity between the first and second concatenated features can be calculated. Simultaneously, based on the positional difference between the image and the text fragment, and the similarity between the first and second concatenated features, the correlation between the image and the text fragment can be calculated. Finally, based on the degree of relevance between the image and each text fragment, the text fragment with the highest degree of relevance to the image can be selected from multiple text fragments to establish an association with the image.

[0172] Furthermore, the method for establishing the association between tables and text fragments is similar to that for establishing the association between images and text fragments. The only difference is that the text describing the table is also input into a multimodal embedding model to obtain the multimodal features of the table. Therefore, this application will not elaborate on how to establish the association between tables and text fragments; please refer to the process for establishing the association between images and text fragments described above.

[0173] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0174] Please refer to Figure 10, which is a structural schematic diagram of a multimodal document slicing device provided in this application. As shown in Figure 10, the multimodal document slicing device includes: an acquisition module 1001, used to acquire a multimodal document, the multimodal document including first modal data and second modal data, the first modal data being text data and including multiple text fragments, and the second modal data being non-text data; a processing module 1002, used to determine a first descriptive text corresponding to the second modal data, the first descriptive text being used to describe the content included in the second modal data; the processing module 1002 is also used to select a first text fragment from multiple text fragments and establish an association relationship with the second modal data based on the correlation between the text fragments in the multiple text fragments and the first descriptive text.

[0175] In one possible implementation, the first descriptive text includes at least one of the following: a title of the second modal data, text included in the second modal data, and a summary text of the second modal data, wherein the summary text is a summary of the content included in the second modal data.

[0176] In one possible implementation, the text included in the second modality data includes text with highlighted markers.

[0177] In one possible implementation, the processing module 1002 is further configured to: select a first text fragment from multiple text fragments and establish an association between it and the second modal data based on the degree of relevance between the text fragments in the multiple text fragments and the first descriptive text, as well as the positions of the multiple text fragments and the second modal data in the multimodal document.

[0178] In one possible implementation, the processing module 1002 is further configured to: determine at least one candidate text segment from multiple text segments based on the positional relationship between the text segments in the multiple segments and the second modal data; and select a first text segment from the at least one candidate text segment and establish an association relationship with the second modal data based on the degree of correlation between the text segments in the at least one candidate text segment and the first descriptive text.

[0179] In one possible implementation, the processing module 1002 is further configured to: determine the weight of the relevance between the text segments in the multiple text segments and the first descriptive text based on the positional relationship between the text segments in the multiple text segments and the second modal data; and select the first text segment from the multiple text segments and establish an association relationship with the second modal data based on the relevance between the text segments in the multiple text segments and the first descriptive text, and the weight of the relevance between the text segments in the multiple text segments and the first descriptive text.

[0180] In one possible implementation, the multiple text fragments include a first text fragment. The processing module 1002 is further configured to: perform feature extraction on the first text fragment and the first descriptive text respectively using a text feature extraction model to obtain a first feature corresponding to the first text fragment and a second feature corresponding to the first descriptive text; perform feature extraction on the first text fragment and the second modal data respectively using a multimodal data feature extraction model to obtain a third feature corresponding to the first text fragment and a fourth feature corresponding to the first descriptive text; and determine the similarity between the first text fragment and the first descriptive text based on the first concatenation feature and the second concatenation feature, wherein the first concatenation feature is obtained based on the first feature and the third feature, and the second concatenation feature is obtained based on the second feature and the fourth feature.

[0181] In one possible implementation, the processing module 1002 is further configured to: establish a relationship between the text segments in the multiple text segments based on the semantics of the text segments in the multiple text segments.

[0182] In one possible implementation, the processing module 1002 is further configured to: extract key information of the text segments from multiple text segments using a first model, wherein the key information is used to indicate the semantic information contained in the text segments; determine the semantic category corresponding to the multiple text segments respectively using the first model based on the key information of the multiple text segments; and establish an association relationship for text segments with the same semantic category among the multiple text segments.

[0183] In one possible implementation, the processing module 1002 is further configured to: segment the first modality data using a second model to obtain multiple text fragments; wherein the first model is a large language model, the second model is a text parsing model, and the number of parameters in the second model is less than the number of parameters in the first model.

[0184] Both the acquisition module 1001 and the processing module 1002 can be implemented in software or in hardware. For example, the implementation of the processing module 1002 will be described below. Similarly, the implementation of the acquisition module 1001 can be referenced from the implementation of the processing module 1002.

[0185] As an example of a software functional unit, the processing module 1002 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the processing module 1002 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0186] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0187] As an example of a hardware functional unit, the processing module 1002 may include at least one computing device, such as a server. Alternatively, the processing module 1002 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0188] The processing module 1002 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1002 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1002 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0189] Please refer to Figure 11, which is a schematic diagram of a computing device provided in this application. The computing device 1100 shown in Figure 11 can be used to execute the multimodal document slicing method provided in this embodiment. As shown in Figure 11, the computing device 1100 includes: a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0190] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 1102 can include pathways for transmitting information between various components of computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0191] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0192] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0193] The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby implementing the multimodal document slicing method described above. That is, the memory 1106 stores instructions for executing the multimodal document slicing method.

[0194] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0195] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0196] Please refer to Figure 12, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing a slicing method for multimodal documents.

[0197] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing the slicing method of multimodal documents. In other words, a combination of one or more computing devices 1100 can jointly execute the instructions for executing the slicing method of multimodal documents.

[0198] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the multimodal document slicing method apparatus. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0199] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. Figure 13 is also a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 13, in computing device cluster 1300, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1106 in computing device 1100B stores instructions for executing the functions of the processing module.

[0200] It should be understood that the functions of computing device 1100A shown in Figure 13 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0201] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0202] Referring to Figure 14, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 2 can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0203] Figure 14 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0204] In one embodiment, the computer-readable storage medium 1400 is provided using a signal bearer medium 1401. The signal bearer medium 1401 may include one or more program instructions 1402, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG2.

[0205] In some examples, signal carrying medium 1401 may include computer-readable medium 1403, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0206] In some embodiments, the signal-bearing medium 1401 may include a computer-recordable medium 1404, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1401 may include a communication medium 1405, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1401 may be transmitted by a wireless communication medium 1405 (e.g., a wireless communication medium conforming to the IEEE 1102.X standard or other transmission protocols).

[0207] One or more program instructions 1402 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1402 conveyed to the computing device via a computer-readable medium 1403, a computer-recordable medium 1404, and / or a communication medium 1405.

[0208] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0209] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0210] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0211] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for slicing multimodal documents, characterized in that, include: Obtain a multimodal document, the multimodal document including first modal data and second modal data, the first modal data being text data and including multiple text fragments, and the second modal data being non-text data; Determine the first description text corresponding to the second modal data, wherein the first description text is used to describe the content included in the second modal data; Based on the degree of relevance between the text fragments among the plurality of text fragments and the first descriptive text, a first text fragment is selected from the plurality of text fragments and an association is established with the second modal data.

2. The method according to claim 1, characterized in that, The first descriptive text includes at least one of the following: the title of the second modal data, the text included in the second modal data, and the summary text of the second modal data, wherein the summary text is a summary of the content included in the second modal data.

3. The method according to claim 2, characterized in that, The text included in the second modal data includes text with highlighted markers.

4. The method according to any one of claims 1-3, characterized in that, The step of selecting a first text segment from the plurality of text segments and establishing an association between it and the second modality data based on the relevance between the text segments and the first descriptive text includes: Based on the relevance of the text fragments among the plurality of text fragments to the first descriptive text, and the positions of the plurality of text fragments and the second modal data in the multimodal document, a first text fragment is selected from the plurality of text fragments to establish an association with the second modal data.

5. The method according to claim 4, characterized in that, The step of selecting a first text segment from the plurality of text segments and establishing an association between it and the second modal data based on the relevance of the plurality of text segments to the first descriptive text, and the positions of the plurality of text segments and the second modal data in the multimodal document, includes: Based on the positional relationship between the text fragments in the plurality of text fragments and the second modal data, at least one candidate text fragment is determined from the plurality of text fragments; Based on the degree of relevance between the text fragments in the at least one candidate text fragment and the first descriptive text, the first text fragment is selected from the at least one candidate text fragment and an association is established with the second modal data.

6. The method according to claim 4, characterized in that, The step of selecting a first text segment from the plurality of text segments and establishing an association between it and the second modal data based on the relevance of the plurality of text segments to the first descriptive text, and the positions of the plurality of text segments and the second modal data in the multimodal document, includes: Based on the positional relationship between the text segments in the plurality of segments and the second modal data, the weights of the relevance between the text segments in the plurality of text segments and the first descriptive text are determined; Based on the degree of relevance between the text fragments in the plurality of text fragments and the first descriptive text, and the weight of the degree of relevance between the text fragments in the plurality of text fragments and the first descriptive text, the first text fragment is selected from the plurality of text fragments to establish an association relationship with the second modal data.

7. The method according to any one of claims 1-6, characterized in that, The plurality of text fragments includes a first text fragment, and the method further includes: By performing feature extraction on the first text fragment and the first descriptive text respectively using a text feature extraction model, the first feature corresponding to the first text fragment and the second feature corresponding to the first descriptive text are obtained; By using a multimodal data feature extraction model, feature extraction is performed on the first text fragment and the second modal data respectively to obtain the third feature corresponding to the first text fragment and the fourth feature corresponding to the first descriptive text; The degree of relevance between the first text fragment and the first descriptive text is determined based on the first splicing feature and the second splicing feature, wherein the first splicing feature is obtained based on the first feature and the third feature, and the second splicing feature is obtained based on the second feature and the fourth feature.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the semantics of the text segments among the multiple text segments, an association relationship is established among the text segments among the multiple text segments.

9. The method according to claim 8, characterized in that, The step of establishing associations among the text segments based on their semantics includes: The first model is used to extract key information from the multiple text segments, and the key information is used to indicate the semantic information contained in the text segments. Based on the key information of the multiple text fragments, the semantic category corresponding to each of the multiple text fragments is determined by the first model; Establish association relationships among text fragments with the same semantic category among the multiple text fragments.

10. The method according to claim 9, characterized in that, The method further includes: The first modal data is segmented using a second model to obtain the multiple text fragments; The first model is a large language model, the second model is a text parsing model, and the number of parameters in the second model is less than the number of parameters in the first model.

11. A slicing device for multimodal documents, characterized in that, include: The acquisition module is used to acquire a multimodal document, which includes first modal data and second modal data. The first modal data is text data and includes multiple text fragments. The second modal data is non-text data. The processing module is used to determine the first description text corresponding to the second modal data, wherein the first description text is used to describe the content included in the second modal data; The processing module is further configured to select a first text segment from the plurality of text segments and establish an association relationship with the second modal data based on the degree of relevance between the text segments and the first descriptive text.

12. The apparatus according to claim 11, characterized in that, The first descriptive text includes at least one of the following: the title of the second modal data, the text included in the second modal data, and the summary text of the second modal data, wherein the summary text is a summary of the content included in the second modal data.

13. The apparatus according to claim 12, characterized in that, The text included in the second modal data includes text with highlighted markers.

14. The apparatus according to any one of claims 11-13, characterized in that, The processing module is further configured to: Based on the relevance of the text fragments among the plurality of text fragments to the first descriptive text, and the positions of the plurality of text fragments and the second modal data in the multimodal document, a first text fragment is selected from the plurality of text fragments to establish an association with the second modal data.

15. The apparatus according to claim 14, characterized in that, The processing module is further configured to: Based on the positional relationship between the text fragments in the plurality of fragments and the second modal data, at least one candidate text fragment is determined from the plurality of text fragments; Based on the degree of relevance between the text fragments in the at least one candidate text fragment and the first descriptive text, the first text fragment is selected from the at least one candidate text fragment and an association is established with the second modal data.

16. The apparatus according to claim 14, characterized in that, The processing module is further configured to: Based on the positional relationship between the text segments in the plurality of segments and the second modal data, the weights of the relevance between the text segments in the plurality of text segments and the first descriptive text are determined; Based on the degree of relevance between the text fragments in the plurality of text fragments and the first descriptive text, and the weight of the degree of relevance between the text fragments in the plurality of text fragments and the first descriptive text, the first text fragment is selected from the plurality of text fragments to establish an association relationship with the second modal data.

17. The apparatus according to any one of claims 11-16, characterized in that, The plurality of text fragments includes a first text fragment, and the processing module is further configured to: By performing feature extraction on the first text fragment and the first descriptive text respectively using a text feature extraction model, the first feature corresponding to the first text fragment and the second feature corresponding to the first descriptive text are obtained; By using a multimodal data feature extraction model, feature extraction is performed on the first text fragment and the second modal data respectively to obtain the third feature corresponding to the first text fragment and the fourth feature corresponding to the first descriptive text; The degree of relevance between the first text fragment and the first descriptive text is determined based on the first splicing feature and the second splicing feature, wherein the first splicing feature is obtained based on the first feature and the third feature, and the second splicing feature is obtained based on the second feature and the fourth feature.

18. The apparatus according to any one of claims 11-17, characterized in that, The processing module is further configured to: Based on the semantics of the text segments among the multiple text segments, an association relationship is established among the text segments among the multiple text segments.

19. The apparatus according to claim 18, characterized in that, The processing module is further configured to: The first model is used to extract key information from the multiple text segments, and the key information is used to indicate the semantic information contained in the text segments. Based on the key information of the multiple text fragments, the semantic category corresponding to each of the multiple text fragments is determined by the first model; Establish association relationships among text fragments with the same semantic category among the multiple text fragments.

20. The apparatus according to claim 19, characterized in that, The processing module is further configured to: The first modal data is segmented using a second model to obtain the multiple text fragments; The first model is a large language model, the second model is a text parsing model, and the number of parameters in the second model is less than the number of parameters in the first model.

21. A computing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the computing device performs the method as described in any one of claims 1 to 10.

22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 10.

23. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 10.

Images