Information processing method and apparatus, device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide an information processing method and apparatus, a device, a storage medium, and a program product. The method comprises: on the basis of a user input, determining at least one document fragment associated with the user input, wherein a target document fragment among the at least one document fragment comprises one or more pieces of media content; generating prompt information on the basis of text in the at least one document fragment and the user input, wherein the prompt information comprises one or more identifiers, and an identifier corresponds to media content among the one or more pieces of media content and is related to the position of the corresponding media content in the target document fragment; and on the basis of the prompt information, using a machine learning model to generate a reply to the user input.

Description

Information processing methods, apparatus, equipment, storage media and program products

[0001] This application claims priority to Chinese Patent Application No. 202411847886.3, filed on December 14, 2024, entitled "Information Processing Method, Apparatus, Device, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to information processing methods, apparatus, devices, computer-readable storage media, and computer program products. Background Technology

[0003] With the development of information technology, various terminal devices can provide people with a variety of services in work and life. Applications providing these services can be deployed on these terminal devices. The terminal devices present relevant content and interact with users through the application's user interface to meet various user needs. In some cases, users may initiate queries or inquiries within the application. Therefore, how to leverage knowledge bases to enable applications to generate higher-quality query results based on user input is a key concern. Summary of the Invention

[0004] In a first aspect of this disclosure, an interaction method is provided. The method includes: determining at least one document fragment associated with user input, the target document fragment of the at least one document fragment including one or more media contents; generating prompt information based on text in the at least one document fragment and the user input, the prompt information including one or more identifiers, one of the identifiers corresponding to media content in the one or more media contents and relating to the position of the corresponding media content in the target document fragment; and generating a response to the user input based on the prompt information using a machine learning model, wherein, if the response includes content of the target text fragment, at least one media content of the one or more media contents is associated with at least a portion of the text of the target text fragment and displayed in the response.

[0005] In a second aspect of this disclosure, an interactive apparatus is provided. The apparatus includes: a document determination module configured to determine at least one document fragment associated with user input based on user input, wherein a target document fragment in the at least one document fragment includes one or more media contents; a prompt information generation module configured to generate prompt information based on text in the at least one document fragment and user input, the prompt information including one or more identifiers, one of the identifiers corresponding to media content in one or more media contents and relating to the position of the corresponding media content in the target document fragment; and a response module configured to generate a response to user input based on the prompt information and using a machine learning model, wherein, if the response includes content of the target text fragment, at least one media content in the one or more media contents is associated with at least a portion of the text of the target text fragment in the response.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The product includes computer-executable instructions, wherein when executed by a processor, the computer-executable instructions implement the method according to a first aspect of this disclosure.

[0009] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 illustrates an example architecture for information processing according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates a flowchart of an example process for constructing a knowledge base according to some embodiments of the present disclosure;

[0014] Figure 4 shows a flowchart of an example process of a question-and-answer process according to some embodiments of the present disclosure;

[0015] Figure 5 shows a flowchart of an information processing method according to some embodiments of the present disclosure;

[0016] Figure 6 illustrates an exemplary structural block diagram of an information processing apparatus according to some embodiments of the present disclosure; and

[0017] Figure 7 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0020] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0027] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0028] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, a target application 112 is installed on a client device 110. A user 130 can interact with the application service component target application 112 via the client device 110 and / or an attached device to the client device 110.

[0029] In some embodiments, the target application 112 may be downloaded and installed on the client device 110. In some embodiments, the target application 112 may also be accessed in other ways, such as through a web page. In some embodiments, in the environment 100 of FIG1, in response to the launch of the target application 112, the client device 110 may present the interface 140 of the target application 112. The interface 140 may be, for example, the interactive interface of the target application 112.

[0030] In some embodiments, the target application 112 may have intelligent dialogue and information processing capabilities. For example, the target application 112 may be a user's digital assistant or an application configured with a digital assistant. In embodiments of this disclosure, the target application 112 is used to interact with the user 130 to assist the user 130 in using a terminal or processing and searching information. In some embodiments, an interaction window with the target application 112 may be presented in the interface 140. In the interaction window, the user 130 can interact with the target application 112 by inputting natural language, images, audio files, video files, web page files, etc., to instruct the target application 112 to assist in completing various tasks, including operations on content entities; or to instruct the target application 112 to complete question-and-answer, query, or search. In some embodiments, the interaction between the user 130 and the target application 112 may include interaction between the user 130 and the digital assistant. For example, the user's input to the target application 112 may be directed to the digital assistant, and the response from the target application 112 to the user 130 may be directed from the digital assistant to the user 130.

[0031] In some embodiments, multiple interaction modes can be provided between the user 130 and the target application 112, and the user can flexibly switch between these modes. When a certain interaction mode is triggered, a corresponding interaction area is presented to facilitate interaction between the user 130 and the target application 112. The interaction methods between the user 130 and the target application 112 differ under different interaction modes, thus flexibly adapting to the interaction needs of different application scenarios.

[0032] In some embodiments, information processing services specific to user 130 can be provided based on historical interaction information between user 130 and target application 112 and / or data ranges specific to user 130. In some embodiments, historical interaction information of user 130 interacting with target application 112 in multiple interaction modes can all be associated with user 130 and stored. Thus, in one of the multiple interaction modes (any or a specified interaction mode), target application 112 can provide services to user 130 based on the historical interaction information associated with user 130 stored therein.

[0033] The target application 112 can be invoked or activated by an appropriate means (e.g., shortcut keys, buttons, or voice) to present an interactive window with the user 130. Selecting the target application 112 opens an interactive window. The interactive window may include interface elements for information interaction, such as input boxes, message lists, message bubbles, etc. In other embodiments, the target application 112 can be invoked through an entry control or menu provided in the interface 140, or by inputting a preset command.

[0034] The interaction window between the target application 112 and the user 130 may include a session window, such as a session window in an instant messaging application or an instant messaging module of a specific application. In the session window, the interaction between the target application 112 and the user 130 may be presented in the form of session messages. Alternatively or additionally, the interaction window between the target application 112 and the user 130 may also include other types of windows, such as a floating window, in which the user 130 can trigger the target application 112 to perform corresponding operations by inputting commands, selecting shortcuts, etc.

[0035] In some embodiments, the target application 112 may support an interactive mode of a conversation window, also known as a conversation mode. In this interactive mode, a conversation window is presented between the user 130 and the target application 112, where the user 130 and the target application 112 interact through conversation messages. In conversation mode, the target application 112 can perform tasks based on the conversation messages in the conversation window. In the interaction window, the user 130 inputs an interaction message, and the target application 112 responds to the user's input by providing a reply message. A conversation window with the target application 112 can be opened by selecting the target application 112. The conversation window may include interface elements for information interaction, such as input boxes, message lists, message bubbles, etc.

[0036] In some embodiments, client device 110 communicates with server 120 to provide services to target application 112. Client device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client device 110 may also support any type of user-facing interface (such as "wearable" circuitry). Server 120 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc. Server 120 may deploy one or more various machine learning models 122 to provide services to target application 112. In some embodiments, model 122 can be a machine learning model, a deep learning model, a learning model, a neural network, etc. In some embodiments, the model may be based on a language model (LM). Language models can acquire question-answering capabilities by learning from large corpora. Model 122 can also be based on other appropriate models.

[0037] In some implementations, the implementation of at least some functions of the target application 112, and / or the implementation of at least some functions of the digital assistant in the target application 112, may be based on models. During the creation or operation of the target application 112, one or more models 122 may be invoked, such as the capabilities of model 122. In the target application 112, the digital assistant may utilize model 122 to understand user input and provide responses to the user based on the output of model 122.

[0038] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0039] As mentioned earlier, a key concern is how to leverage knowledge bases to enable applications to generate higher-quality query results based on user input. In intelligent question answering, current general-purpose mechanisms lack the ability to retrieve media content (such as images). However, typical question-and-answer scenarios, such as customer service hotline numbers and official website help centers, have a strong demand for including relevant images in the responses.

[0040] Currently, when using models, especially language models, to handle intelligent question-answering scenarios with media content retrieval requirements, the following problems exist. First, when generating text content, large models frequently misreference media content (e.g., images), using irrelevant or incorrect media content as the final generated answer. This is usually due to misunderstandings in associating text descriptions with media content, or overly complex prompts that prevent the model from fully following and understanding the instructions. Second, when using language models to generate text content, instability often occurs when referencing or generating media content links (e.g., Uniform Resource Locator URLs) in a predefined format (e.g., Markdown format). This is mainly manifested in the model frequently generating incorrect or inaccurate URLs when restating or referencing media content links. This instability may be due to inconsistencies in URLs in the model's training data, or the model failing to correctly... Accurate understanding and processing of the context of linked text is crucial. Incorrect URL generation leads to inaccurate content, affecting user trust in the generated content, especially in professional fields requiring precise document and data retrieval, such as law and academic research. Third, when processing text in predefined formats (e.g., Markdown), media content is often referenced via long public Hypertext Transfer Protocol (HTTP) links or FileIDs. These long links occupy significant space within text slices, reducing the density of effective information. When this text is converted into vectors for information retrieval, the presence of numerous non-semantic elements (such as URLs) often results in vectors that fail to accurately reflect the text's true semantic content. This phenomenon impacts the accuracy of vector-based retrieval systems.

[0041] In view of the above, according to embodiments of this disclosure, an improved information processing scheme is provided. Specifically, at least one document fragment associated with user input is determined based on user input, wherein the target document fragment in the at least one document fragment includes one or more media contents. Based on the text in the at least one document fragment and the user input, a prompt message is generated. The prompt message includes one or more identifiers, wherein one of the identifiers corresponds to media content in the one or more media contents and is related to the position of the corresponding media content in the target document fragment. Based on the prompt message, a response to the user input is generated using a machine learning model. When the response includes the content of the target text fragment, at least one media content in the one or more media contents is associated with at least a portion of the text of the target text fragment and displayed in the response.

[0042] According to the scheme of this disclosure, an identifier is used to replace media content in the document when generating prompt information. Compared with media content, the identifier is more concise and easier to understand. This allows the machine learning model to accurately reference the identifier when generating output, ultimately achieving accurate media content reference in the response to user input. Therefore, in intelligent question-answering scenarios with media content retrieval requirements, media content can be accurately retrieved and correctly referenced in responses, thereby providing users with higher quality query results or question-answering results. When media content is included in the response, since the media content is displayed in association with the corresponding text content, the media content included in the response can be placed in the appropriate or correct position. This improves the consistency between media content and text in the response, further improving the quality of the response. Furthermore, displaying media content and corresponding text in association helps users understand the response. The embodiments of this disclosure can accurately place media content in the response, thereby avoiding interference to users caused by incorrect insertion of media content.

[0043] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0044] Figure 2 illustrates an example architecture 200 for information processing according to some embodiments of the present disclosure. As shown in Figure 2, the example architecture 200 includes a knowledge base 210, a target application 112, and one or more machine learning models 122. The knowledge base 210 is used to store document data required to provide responses to user input, such as user queries, inquiries, or searches. In the knowledge base 210, the document data is stored in the form of vectorized document fragments, thereby facilitating efficient data retrieval.

[0045] In some embodiments, the knowledge base 210 stores only the text content of document data, and not the media content in the original document corresponding to the document data, such as images, charts, audio, video, structured cards, etc. This increases the density of effective information in the knowledge base 210, so that when retrieving the required document data from the knowledge base 210, interference from media content can be avoided, thus improving the accuracy of text retrieval. Detailed information on the construction of the knowledge base will be described later in conjunction with Figure 3, and will not be elaborated upon here.

[0046] Target application 112 retrieves one or more document fragments associated with user input 220 from knowledge base 210. Target application 112 then generates context information based on these document fragments. In some embodiments, the original document fragment corresponding to the target document fragment among the one or more document fragments associated with the user input includes one or more media contents. In this case, target application 112 restores the one or more media contents to the target document fragment using identifiers; specifically, target application 112 adds identifiers corresponding to the one or more media contents to the text of the target document fragment. The position of the identifiers corresponding to the one or more media contents corresponds to the position of the one or more media contents in the original document fragment.

[0047] The target application 112 generates context information based on one or more document fragments, and then generates prompt information based on the context information and user input. In some embodiments, the original document fragment corresponding to the target document fragment in one or more document fragments associated with the user input includes one or more media contents. In this case, since the one or more document fragments have undergone the aforementioned processing, the prompt information also includes one or more identifiers. The identifiers in the one or more identifiers correspond to the media contents in the one or more media contents and are related to the position of the corresponding media contents in the target document fragment. Details regarding adding identifiers to the target document fragment will be described later in conjunction with Figure 4, and will not be repeated here.

[0048] In embodiments of this disclosure, the prompts are simplified by replacing media content in document fragments with identifiers, and the identifiers accurately associate text descriptions with media content. Thus, when the machine learning model generates a response based on user input and prompts, it no longer needs to reference or generate media content links in a predetermined format (e.g., Markdown format), thereby improving the accuracy of the content generated by the machine learning model.

[0049] The target application 112 provides the prompt information to the machine learning model 122 and uses the machine learning model 122 to generate a response 230 to the user input 220. In a question-and-answer scenario, the response 230 to the user input is also called an answer. In a query scenario, the response 230 to the user input is also called a query result.

[0050] In some embodiments, an example process for the target application 112 to generate a response 230 to user input 220 is as follows. First, the target application 112 provides a prompt to the machine learning model 122 to obtain the output of the machine learning model 122. Then, in response to determining that the output of the machine learning model 122 includes a first identifier from one or more identifiers, the target application 112 obtains the media content corresponding to the first identifier. Then, the target application 112 adds the obtained media content to the response 230 to user input 220. Details regarding the addition of media content in the response 230 will be described later with reference to Figure 4, and will not be repeated here.

[0051] Figure 3 illustrates a flowchart of an example process 300 for constructing a knowledge base according to some embodiments of the present disclosure. For ease of discussion, process 300 will be described with reference to the environment 100 of Figure 1. Process 300 may be implemented at client device 110 and / or server 120. For ease of description, it will be illustrated by assuming that process 300 is implemented at server 120.

[0052] It should be noted that, taking the implementation of process 200 at client device 110 as an example, some operations described with reference to client device 110 may require the assistance of server 120 to complete. It should also be noted that the operations performed by client device 110 may specifically be performed by relevant applications and / or target applications installed on client device 110.

[0053] In box 310, server 120 obtains and parses documents used to build the knowledge base. In some embodiments, the documents used to build the knowledge base may be obtained by server 120 from a public database or from the Internet based on user input or other information. In some embodiments, the documents used to build the knowledge base may be internal document data of an entity or organization (e.g., a company or enterprise), and server 120 may obtain the documents used to build the knowledge base in response to data uploaded by the user. That is, obtaining the documents used to build the knowledge base may be achieved by server 120 based on public data searches, or it may be provided by the user.

[0054] In some embodiments, the target document in the documents used to build the knowledge base includes one or more media contents. After the server 120 obtains the target document, it parses the target document, retrieves and stores one or more media contents within it. For example, the server 120 stores one or more media contents in the target document at a predetermined storage location. The predetermined storage location can be local storage or cloud storage. In some embodiments, the server 120 identifies the one or more media contents using a resource identifier (UID). Thus, when a user's response requires referencing media content from the one or more media contents, the media content can be retrieved using its resource identifier and added to the response.

[0055] In box 320, server 120 divides the documents used to build the knowledge base into multiple document fragments. For example, the documents used to build the knowledge base can be divided into multiple document fragments based on the semantics of the documents used to build the knowledge base or other criteria, using text segmentation methods commonly used in the art.

[0056] In some embodiments, a target document fragment among multiple document fragments includes one or more media contents. For each target document fragment, server 120 stores the corresponding information of the media contents within the one or more media contents included in the target document fragment into the target document fragment's metadata, to record the corresponding position information and corresponding resource identifier of the media contents within the target document fragment. The corresponding position information of the media contents within the one or more media contents within the target document fragment can be obtained by parsing the target document fragment or the original document corresponding to the target document fragment.

[0057] In box 330, server 120 removes media content from multiple document fragments, resulting in updated document fragments. In this disclosure, an updated document fragment refers to a document fragment obtained after removing media content, such as images, from the original document fragment; that is, the text within the document fragment. When a document fragment does not contain media content, the original document fragment and the updated document fragment are identical. In this disclosure, because the media content included in the document fragment is removed, the updated document fragment does not contain media content links in a predetermined format (e.g., Markdown format), increasing the effective information density of the document fragment. Therefore, when subsequently recalling document fragments (i.e., the updated document fragments), the generated text reflects the true semantics because non-semantic elements (e.g., URLs) are excluded. This improves the accuracy of document fragment recall. It should be noted that the recalled document fragment refers to the updated document fragment.

[0058] In box 340, server 120 stores the corresponding vectorized representations of a plurality of updated document fragments. In some embodiments, server 120 may use a predetermined embedding model to convert the updated plurality of document fragments into a vector matrix. In some embodiments, server 120 stores the corresponding vectorized representations of the updated plurality of document fragments in a predetermined knowledge path, such as knowledge base 210 in FIG2.

[0059] Figure 4 illustrates a flowchart of an example process 400 for information processing according to some embodiments of the present disclosure. For ease of discussion, process 400 will be described with reference to the environment 100 of Figure 1. Process 400 may be implemented at client device 110 and / or server 120. For ease of description, it will be illustrated by assuming that process 400 is implemented at server 120.

[0060] It should be noted that, taking the implementation of process 400 at client device 110 as an example, some operations described with reference to client device 110 may require the assistance of server 120 to complete. It should also be noted that the operations performed by client device 110 may specifically be performed by relevant applications and / or target applications installed on client device 110.

[0061] In box 410, server 120 receives user input. For example, user input provided by a user at client device 110 can be sent to server 120. In box 420, server 120 determines at least one updated document fragment associated with the user input based on the user input. In some embodiments, server 120 retrieves at least one updated document fragment associated with the user input from a knowledge base using a predetermined retrieval method, such as similarity retrieval or full-text retrieval.

[0062] In box 430, server 120 uses identifiers to reconstruct one or more media contents to at least one updated document fragment to generate contextual information. In box 440, server 120 generates prompt information based on user input and contextual information. That is, server 120 injects the recalled document fragment into a machine learning model to generate prompt information.

[0063] In some embodiments, the target document fragment corresponding to the updated target document fragment in at least one updated document fragment determined in block 420 includes one or more media contents. In this case, server 120 uses an identifier to restore one or more media contents to the updated target document fragment.

[0064] In some embodiments, the identifier can be an identifier with a predetermined format. Taking an image as an example, the identifier with a predetermined format could be, for example, "illustration: <<image x-x> >”, where “illustration” is used to indicate that there is an image in the model context. The image identifies the media content type as an image. The first x represents the image's index in at least one updated document fragment within the updated target document fragment. The second x represents the image's index in one or more images within the target document fragment corresponding to the updated target document fragment. For example, server 120 determines two updated document fragments associated with user input. The first updated document fragment (i.e., the first updated target document fragment) corresponds to the first document fragment (i.e., the first target document fragment) which includes two images; then the first image could be representing <<image 1-1> The second image can be represented as <<image 1-2> The second updated document fragment (i.e., the second updated target document fragment) contains three images, which can be represented as < <image2-1>>、<<image 2-2> > and <<image 2-3> It should be understood that while this disclosure uses images as examples to illustrate identifiers of a predetermined format, this disclosure is not limited to images.

[0065] In some embodiments, an example process for restoring one or more media contents to at least one updated document fragment using identifiers to generate context information is as follows: Server 120 obtains the corresponding location information of one or more media contents from the metadata of the target document fragment corresponding to the updated target document fragment. Then, based on the corresponding location information of one or more media contents, server 120 adds the corresponding identifiers of one or more media contents in a predetermined format to the updated target document fragment to generate context information. If the first updated document fragment (i.e., the first updated target document fragment) includes two images, then based on the corresponding location information of the first image in the first document fragment, "Illustration: <<image 1-1> Add ">" to the first updated document fragment. For example, if the first image is after the fifth line of the first document fragment, then add "Illustration:<<image 1-1> >” is added after the fifth line of the first updated document fragment.

[0066] After restoring one or more media contents to the updated target document fragment using identifiers in a predefined format, server 120 generates prompt information based on user input and context information. The prompt information guides the model to generate the correct response based on user input and context information, and, if the target document fragment includes media content, appends the relevant media content and outputs the media content using identifiers. This significantly reduces the probability of the machine learning model generating incorrect media content because the identifiers are much shorter than links to media content in formats such as Markdown.

[0067] Example prompts and model outputs are shown in Table 1.

[0068] Table 1

[0069] The upper part of Table 1 shows examples of system prompts from the model. The lower part shows examples of the model's output. It should be understood that the above prompts and model outputs are merely examples and do not constitute a limitation of this disclosure. Prompts may also include more or less content than those shown. For example, prompts may also include information related to model roles, glossary, model language style, output goals, or output constraints. In some embodiments, prompts may include instructions that, when the model's output includes identifiers, the identifiers need to be associated with the corresponding content output, for example, the identifier immediately following the relevant content output. As shown in Table 1, the identifier corresponding to the lion image immediately follows the general description of the lion's habits, rather than following the description of the differences in habits between lions and giant pandas. The identifier corresponding to the giant panda image immediately follows the description of the differences in habits between lions and giant pandas, rather than being located in the lion description section. This restriction, on the one hand, accurately restores the position of images (which are examples of media content) in the document, and on the other hand, facilitates higher media content-text consistency in subsequent responses generated based on the model output. Taking Table 1 as an example, in the responses generated based on the model output, the image of a lion is displayed in association with an overview of the lion's lifestyle, for example, the image of a lion follows the overview of the lion's lifestyle.

[0070] In some embodiments, the prompt information may include constraint information. This constraint information may, for example, instruct the model not to include graphic symbols in the output identifiers, or not to output media content in some markup language or programming language, such as not to output images in Markdown format.

[0071] In some embodiments, the identifier can be a sentence identifier. The sentence identifier identifies a sentence in at least one updated document fragment. The sentence identifier is, for example, a sentence ID. After server 120 determines at least one updated document fragment associated with user input in box 420, the prompt message generation process is as follows: Server 120 divides the at least one updated document fragment into multiple sentences at the sentence level. Then, server 120 generates a sentence identifier for each of the multiple sentences based on the order of the at least one updated document fragment and the order of the multiple sentences. Then, server 120 determines the prompt message based on the sentence identifiers generated for the multiple sentences, the at least one updated document fragment, and the user input. In some embodiments, for example, if the at least one updated document fragment includes four sentences, identifiers representing the first, second, third, and fourth sentences can be generated for each of the four sentences, such as sentence numbers.

[0072] In some embodiments, the target document fragment corresponding to the updated target document fragment in at least one updated document fragment includes one or more media contents, and the server 120 also associates the one or more media contents with the sentence identifier of the updated target document fragment. An example association process is as follows: For a given media content in one or more media contents, the server 120 obtains the location information and resource identifier of the given media content from the metadata of the target document fragment. Then, based on the location information of the given media content, the server 120 determines the first sentence in the updated target document fragment adjacent to the given media content, such as the first adjacent sentence before the given media content, or the first adjacent sentence after the given media content. The server 120 then associates the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0073] Referring again to Figure 4, in box 450, server 120 uses a machine learning model to generate a response to the user input based on the prompt information.

[0074] In some embodiments, the identifier is an identifier with a predetermined format. In this case, an example process for generating a response to user input is as follows: Server 120, in response to a machine learning model output including a second identifier with a predetermined format provided based on prompt information, retrieves the media content corresponding to the second identifier. Server 120 then replaces the second identifier in the model output with the retrieved media content to update the model output. Server 120 then determines a response based on the updated model output. In this scheme, the predetermined format identifier allows the model to accurately generate references to the media content, and by removing the predetermined format media content information, text recall is improved. Furthermore, the original text can be losslessly restored in the final response using the corresponding location information and resource identifier of the media content stored in the metadata.

[0075] In some embodiments, where the response includes target media content, the media content and its corresponding text are displayed in association within the response. For example, the media content may immediately follow the corresponding text. Alternatively, the media content may be displayed side-by-side with the corresponding text. This improves the consistency between the media content and the text in the response.

[0076] In some embodiments, an example process for obtaining media content corresponding to the second identifier is as follows: Server 120 determines a resource identifier from the metadata of the target document fragment based on the second identifier. Then, server 120 uses the determined resource identifier to obtain the media content corresponding to the second identifier.

[0077] In some embodiments, the identifier is a sentence identifier with a predetermined format. In this case, an example process for generating a response to user input is as follows: Server 120 provides prompt information to machine learning model 122 to obtain the model output of machine learning model 122. Then, in response to determining that the second sentence identifier in the model output is associated with a resource identifier, server 120 obtains the media content corresponding to the second sentence identifier based on the associated resource identifier. Server 120 then adds the obtained media content to the position adjacent to the second sentence identifier to generate a response to user input. The process of obtaining the media content corresponding to the second sentence identifier based on the associated resource identifier is described above and will not be repeated here. In this scheme, by reducing the large model's understanding of media content (e.g., images) in the context, the task of outputting image quotations is transformed into a task of outputting text quotations, which the model is better at, thereby reducing the probability of model errors and improving the overall system stability.

[0078] Figure 5 illustrates a flowchart of an information processing procedure 500 according to some embodiments of the present disclosure. For ease of discussion, the procedure 500 will be described with reference to the environment 100 of Figure 1. The procedure 500 may be implemented at a server 120 or a client device 110, or may be implemented by the server 120 in conjunction with the client device 110.

[0079] In box 510, server 120 determines at least one document fragment associated with user input based on user input, wherein the target document fragment in the at least one document fragment includes one or more media contents.

[0080] In box 520, server 120 generates a prompt message based on text in at least one document fragment and user input. The prompt message includes one or more identifiers, one of which corresponds to media content in one or more media contents and is related to the position of the corresponding media content in the target document fragment.

[0081] In box 530, based on the prompt information, a machine learning model is used to generate a response to the user input, wherein if the response includes the content of the target text fragment, at least one of one or more media contents is associated with at least a portion of the text of the target text fragment and displayed in the response.

[0082] In some embodiments, process 500 further includes: providing a prompt to a machine learning model to obtain the output of the machine learning model; in response to determining that the output includes a first identifier from one or more identifiers, obtaining media content corresponding to the first identifier; and adding the obtained media content to a response.

[0083] In some embodiments, the identifier in one or more identifiers includes an identifier in a predetermined format, which includes at least one of the following: information indicating the type of the corresponding media content, the sequence number of the target document fragment in at least one document fragment, and the sequence number of the corresponding media content in one or more media contents.

[0084] In some embodiments, generating prompt information includes: obtaining the corresponding location information of one or more media contents from the metadata of the target document fragment; adding the corresponding identifiers of one or more media contents in a predetermined format to the text of the target document fragment based on the corresponding location information of the one or more media contents to generate context information; and determining the prompt information based on the context information and user input.

[0085] In some embodiments, generating a response to user input includes: in response to a model output provided by a machine learning model based on prompt information, which includes a second identifier in a predetermined format, obtaining media content corresponding to the second identifier; replacing the second identifier in the model output with the obtained media content to update the model output; and determining a response based on the updated model output.

[0086] In some embodiments, obtaining the media content corresponding to the second identifier includes: determining a resource identifier from the metadata of the target document fragment based on the second identifier; and obtaining the media content corresponding to the second identifier using the determined resource identifier.

[0087] In some embodiments, the identifiers among one or more identifiers include sentence identifiers, which identify sentences in the target document fragment.

[0088] In some embodiments, generating prompt information includes: dividing at least one document segment into multiple sentences at the sentence level; generating a sentence identifier for each of the multiple sentences based on the order of the at least one document segment and the order of the multiple sentences; and determining prompt information based on the sentence identifiers generated for the multiple sentences respectively, the text in the at least one document segment, and user input.

[0089] In some embodiments, process 500 further includes: for a given media content in one or more media contents, obtaining location information and resource identifier of the given media content from the metadata of a target document fragment; determining a first sentence in the target document fragment adjacent to the given media content based on the location information of the given media content; and associating the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0090] In some embodiments, generating a response to user input includes: providing a prompt to a machine learning model to obtain the model output of the machine learning model; in response to determining that a second sentence identifier in the model output is associated with a resource identifier, obtaining media content corresponding to the second sentence identifier based on the associated resource identifier; and adding the obtained media content to a position adjacent to the second sentence identifier to generate a response.

[0091] In some embodiments, process 500 further includes: dividing the original document including the target document fragment into a plurality of document fragments, the target document fragment being one of the plurality of document fragments; removing media content included in the plurality of document fragments to obtain updated plurality of document fragments; and storing corresponding vectorized representations of the updated plurality of document fragments.

[0092] In some embodiments, process 500 further includes: determining the corresponding location information and corresponding resource identifier of one or more media contents in the target document fragment; and storing the corresponding location information and corresponding resource identifier of one or more media contents in the metadata corresponding to the target document fragment.

[0093] The solution disclosed herein reduces the error rate of the model referencing media content to, for example, 2.5%, and increases the end-to-end accuracy from, for example, 20.6% to, for example, 92.5%. The solution disclosed herein eliminates the occurrence of media content URLs with incorrect predefined formats (e.g., Markdown). The solution disclosed herein significantly improves recall, achieving a 95% recall rate for segments relevant to user input. The solution disclosed herein solves the current problems related to media content recall in intelligent question answering without introducing a large multimodal model, increasing the probability that the application can generate higher-quality responses to user input through a knowledge base.

[0094] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 6 shows an exemplary structural block diagram of an interactive apparatus 600 according to some embodiments of this disclosure. Apparatus 600 may be implemented as or included in client device 110 and / or server 120. The various modules / components in apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0095] As shown in Figure 6, the device 600 includes a document determination module 610, configured to determine at least one document fragment associated with user input based on user input, wherein the target document fragment in the at least one document fragment includes one or more media contents. The device 600 includes a prompt information generation module 620, configured to generate prompt information based on text in the at least one document fragment and user input, the prompt information including one or more identifiers, where one of the identifiers corresponds to media content in one or more media contents and is related to the position of the corresponding media content in the target document fragment. The device 600 includes a response module 630, configured to generate a response to user input based on the prompt information and using a machine learning model, wherein when the response includes content from the target text fragment, at least one media content from the one or more media contents is associated with at least a portion of the text of the target text fragment and displayed in the response.

[0096] In some embodiments, the response module 630 is further configured to: provide a prompt message to a machine learning model to obtain the output of the machine learning model; in response to determining that the output includes a first identifier from one or more identifiers, obtain media content corresponding to the first identifier; and add the obtained media content to the response.

[0097] In some embodiments, the identifier in one or more identifiers includes an identifier in a predetermined format, which includes at least one of the following: information indicating the type of the corresponding media content, the sequence number of the target document fragment in at least one document fragment, and the sequence number of the corresponding media content in one or more media contents.

[0098] In some embodiments, the prompt information generation module 620 is further configured to: obtain the corresponding location information of one or more media contents from the metadata of the target document fragment; add the corresponding identifiers of one or more media contents in a predetermined format to the text of the target document fragment based on the corresponding location information of the one or more media contents to generate context information; and determine the prompt information based on the context information and user input.

[0099] In some embodiments, the response module 630 is further configured to: in response to a model output provided by a machine learning model based on prompt information, including a second identifier in a predetermined format, acquire media content corresponding to the second identifier; replace the second identifier in the model output with the acquired media content to update the model output; and determine a response based on the updated model output.

[0100] In some embodiments, the response module 630 is further configured to: determine a resource identifier from the metadata of the target document fragment based on the second identifier; and use the determined resource identifier to obtain media content corresponding to the second identifier.

[0101] In some embodiments, the identifiers among one or more identifiers include sentence identifiers, which identify sentences in the target document fragment.

[0102] In some embodiments, the prompt message generation module 620 is further configured to: divide at least one document fragment into multiple sentences at the sentence granularity; generate a sentence identifier for each of the multiple sentences based on the order of the at least one document fragment and the order of the multiple sentences; and determine a prompt message based on the sentence identifiers generated for the multiple sentences respectively, the text in the at least one document fragment, and user input.

[0103] In some embodiments, the prompt information generation module 620 is further configured to: for a given media content in one or more media contents, obtain the location information and resource identifier of the given media content from the metadata of the target document fragment; determine the first sentence in the target document fragment adjacent to the given media content based on the location information of the given media content; and associate the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

[0104] In some embodiments, the response module 630 is further configured to: provide a prompt message to a machine learning model to obtain the model output of the machine learning model; in response to determining that the second sentence identifier in the model output is associated with a resource identifier, obtain media content corresponding to the second sentence identifier based on the associated resource identifier; and add the obtained media content to a position adjacent to the second sentence identifier to generate a response.

[0105] In some embodiments, the apparatus 600 further includes an index creation module configured to: divide an original document including a target document fragment into a plurality of document fragments, the target document fragment being one of the plurality of document fragments; remove media content included in the plurality of document fragments to obtain updated plurality of document fragments; and store corresponding vectorized representations of the updated plurality of document fragments.

[0106] In some embodiments, the index creation module is further configured to: determine the corresponding location information and corresponding resource identifier of one or more media contents in the target document fragment; and store the corresponding location information and corresponding resource identifier of one or more media contents in the metadata corresponding to the target document fragment.

[0107] The units and / or modules included in device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0108] It should be understood that one or more steps in the above methods can be performed by appropriate electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may, for example, include devices running the system management platform 110 in Figure 1.

[0109] Figure 7 illustrates a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 700 shown in Figure 7 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 700 shown in Figure 7 may include or be implemented as the system management platform 110 of Figure 1 or the device 600 of Figure 6.

[0110] As shown in Figure 7, the electronic device 700 is in the form of a general-purpose electronic device. Components of the electronic device 700 may include, but are not limited to, one or more processing units or processors 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processor 710 may be a physical or virtual processor and is capable of performing various processes according to the programs stored in the memory 720. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0111] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0112] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0113] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0114] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0115] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0116] According to an exemplary implementation of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments of FIG5; therefore, further details will not be provided here.

[0117] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0118] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0119] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0121] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Based on user input, at least one document fragment is determined to be associated with the user input, wherein the target document fragment in the at least one document fragment includes one or more media contents; Based on the text in the at least one document fragment and the user input, a prompt message is generated, the prompt message including one or more identifiers, wherein one of the one or more identifiers corresponds to the media content in the one or more media content and is related to the position of the corresponding media content in the target document fragment; as well as Based on the prompt information, a machine learning model is used to generate a response to the user input, wherein when the response includes the content of the target text fragment, at least one of the one or more media contents is associated with at least a portion of the text of the target text fragment in the response.

2. The method of claim 1, wherein generating a response to the user input comprises: The prompt information is provided to the machine learning model to obtain the output of the machine learning model; In response to determining that the output includes a first identifier among the one or more identifiers, the media content corresponding to the first identifier is obtained; as well as Add the obtained media content to the response.

3. The method according to any one of claims 1 to 2, wherein the identifier in the one or more identifiers includes an identifier of a predetermined format, the identifier of the predetermined format including at least one of the following: Information indicating the type of media content. The sequence number of the target document fragment in the at least one document fragment, or The corresponding media content's sequence number within the one or more media contents.

4. The method according to claim 3, wherein generating the prompt information includes: Obtain the corresponding location information of the one or more media contents from the metadata of the target document fragment; Based on the corresponding location information of the one or more media contents, the corresponding identifiers of the one or more media contents in a predetermined format are added to the text of the target document fragment to generate context information; as well as The prompt information is determined based on the context information and the user input.

5. The method of claim 3, wherein generating a response to the user input comprises: In response to the machine learning model output provided based on the prompt information including the second identifier in the predetermined format, the media content corresponding to the second identifier is obtained; Replace the second identifier in the model output with the acquired media content to update the model output; as well as The response is determined based on the updated model output.

6. The method according to claim 5, wherein obtaining the media content corresponding to the second identifier comprises: Based on the second identifier, a resource identifier is determined from the metadata of the target document fragment; as well as Using the determined resource identifier, obtain the media content corresponding to the second identifier.

7. The method according to any one of claims 1 to 6, wherein the identifier among the one or more identifiers includes a sentence identifier that identifies a sentence in the target document fragment.

8. The method according to claim 7, wherein generating the prompt information comprises: The at least one document fragment is divided into multiple sentences at the sentence level; Based on the order of the at least one document fragment and the order of the plurality of sentences, generate the sentence identifier for each of the plurality of sentences; as well as The prompt information is determined based on the sentence identifiers generated for the plurality of sentences, the text in the at least one document segment, and the user input.

9. The method according to claim 8, further comprising: For a given media content among the one or more media contents, Obtain the location information and resource identifier of the given media content from the metadata of the target document fragment; Based on the location information of the given media content, determine the first sentence in the target document segment that is adjacent to the given media content; as well as Associate the resource identifier of the given media content with the first sentence identifier of the determined first sentence.

10. The method of claim 7, wherein generating a response to the user input comprises: The prompt information is provided to the machine learning model to obtain the model output of the machine learning model; In response to determining that the second sentence identifier in the model output is associated with a resource identifier, the media content corresponding to the second sentence identifier is obtained based on the associated resource identifier; as well as The acquired media content is added to the position adjacent to the second sentence identifier to generate the response.

11. The method according to any one of claims 1 to 10, further comprising: The original document, including the target document fragment, is divided into multiple document fragments, and the target document fragment is one of the multiple document fragments; Remove the media content included in the plurality of document fragments to obtain updated plurality of document fragments; and Store the corresponding vectorized representations of the updated multiple document fragments.

12. The method of claim 11, further comprising: Determine the corresponding location information and corresponding resource identifier of the one or more media contents in the target document fragment; as well as The corresponding location information and corresponding resource identifier of the one or more media contents are stored in the metadata corresponding to the target document fragment.

13. An information processing apparatus, comprising: The document determination module is configured to determine at least one document fragment associated with the user input based on the user input, wherein the target document fragment in the at least one document fragment includes one or more media contents; The prompt information generation module is configured to generate prompt information based on the text in the at least one document fragment and the user input. The prompt information includes one or more identifiers, wherein one of the one or more identifiers corresponds to the media content in the one or more media contents and is related to the position of the corresponding media content in the target document fragment. as well as The response module is configured to generate a response to the user input based on the prompt information using a machine learning model, wherein when the response includes the content of the target text fragment, at least one of the one or more media contents is associated with at least a portion of the text of the target text fragment in the response.

14. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processor.

15. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 12.

Images