Agent communication interaction method, electronic device and computer-readable storage medium

By generating trigger signals and analyzing multimodal conversation content using historical dialogue data, the problem of low multi-agent collaboration efficiency in multimodal interaction systems is solved, enabling more natural and personalized multimodal interaction.

WO2026232908A1PCT designated stage Publication Date: 2026-11-12ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/078729
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-06
Filing Date
2026-02-11
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing multimodal interaction systems lack efficient collaboration among multiple agents, failing to meet diverse scenario requirements and personalized needs.

Method used

Trigger signals are generated based on the current input multimodal information, multimodal conversation content is determined using pre-stored historical dialogue data, conversation analysis is performed on the multimodal conversation content to obtain the sequence of actions to be executed, and these actions are assigned to multiple agents for execution. Finally, the execution results are fed back to the target object.

Benefits of technology

It enables efficient collaboration among cross-modal multi-agents, providing more natural, efficient, and personalized multimodal interactions, and meeting diverse scenario and personalized needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026078729_12112026_PF_FP_ABST
    Figure CN2026078729_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An agent communication interaction method, an electronic device and a computer-readable storage medium, which relate to the technical fields of artificial intelligence and large models. The method comprises: on the basis of currently input multimodal information, generating a triggering signal, wherein the multimodal information is determined on the basis of a multimodal interaction mode of a target object; using the triggering signal and pre-stored historical conversation data to determine multimodal session content; performing session analysis on the multimodal session content, so as to obtain a sequence of actions to be executed; dispatching the sequence of said actions to a plurality of agents for execution, so as to obtain an execution result; and feeding the execution result back to the target object. The method solves the technical problem of a multimodal interaction system lacking efficient collaboration among a plurality of agents and being incapable of meeting diverse scenario requirements and personalized requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Intelligent agent communication and interaction methods, electronic devices and computer-readable storage media Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and large model technology, and more specifically, to an intelligent agent communication and interaction method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of large-scale models and multimodal technologies, the application of artificial intelligence (AI) hardware is ushering in a new wave of popularity. Products such as smart cockpits and wearable devices have significantly improved the interactive experience by integrating multimodal large-scale models.

[0003] Currently, traditional multimodal interaction systems mostly employ end-to-end speech modeling schemes, achieving a certain level of multimodal understanding and interaction by jointly pre-training data from multiple modalities. However, end-to-end speech modeling schemes generally lack tool usage capabilities and deep customization functions, making it difficult to cope with complex and ever-changing scenario requirements and provide personalized services. Furthermore, while full-duplex speech modeling schemes have been proposed, improving real-time interactivity, they lack the ability to process other modal information such as video, limiting their application breadth in multimodal environments and failing to meet diverse scenario and user needs.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides an intelligent agent communication and interaction method, an electronic device, and a computer-readable storage medium to at least solve the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple intelligent agents and cannot meet diverse scenario requirements and personalized needs.

[0006] According to one aspect of the present disclosure, an intelligent agent communication interaction method is provided, comprising: generating a trigger signal based on currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; determining multimodal conversation content using the trigger signal and pre-stored historical dialogue data; performing conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed; assigning the sequence of actions to be executed to multiple intelligent agents for execution to obtain execution results; and feeding back the execution results to the target object.

[0007] According to another aspect of the embodiments of this disclosure, an intelligent agent communication interaction method is also provided, comprising: generating a trigger signal based on a currently input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction mode of the target object; determining the multimodal merchant recommendation session content using the trigger signal and pre-stored historical merchant recommendation dialogue data; performing session analysis on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence; distributing the merchant recommendation action sequence to multiple intelligent agents for execution to obtain a merchant recommendation execution result; and feeding back the merchant recommendation execution result to the target object.

[0008] According to another aspect of the embodiments of this disclosure, an intelligent agent communication interaction method is also provided, comprising: obtaining an intelligent agent communication interaction request through a first application programming interface, wherein the request data carried in the intelligent agent communication interaction request includes: multimodal information, the multimodal information being determined according to the multimodal interaction mode of the target object; and returning an intelligent agent communication interaction response through a second application programming interface, wherein the response data carried in the intelligent agent communication interaction response includes: an execution result, the execution result being obtained according to any of the above-mentioned intelligent agent communication interaction methods.

[0009] According to another aspect of the embodiments of this disclosure, an intelligent agent communication interaction method is also provided, comprising: acquiring a currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, the multimodal information being determined according to the multimodal interaction mode of the target object; responding to the intelligent agent communication interaction dialogue request, returning an intelligent agent communication interaction dialogue response, wherein the information carried in the intelligent agent communication interaction dialogue response includes: execution result, the execution result being obtained according to any of the above-mentioned intelligent agent communication interaction methods; and displaying the execution result in a graphical user interface.

[0010] According to another aspect of the embodiments of this disclosure, an intelligent agent communication interaction method is also provided, including: responding to an input command applied to an operation interface and displaying multimodal information on the operation interface; responding to a processing command applied to the operation interface and displaying an execution result on the operation interface; wherein the execution result is obtained according to any of the above-described intelligent agent communication interaction methods.

[0011] According to another aspect of the embodiments of this disclosure, an intelligent agent communication interaction system is also provided, comprising: a perception layer, configured to generate a trigger signal based on currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; a control layer, configured to determine multimodal conversation content using the trigger signal and pre-stored historical dialogue data; a planning layer, configured to perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed; an execution layer, configured to assign the sequence of actions to be executed to multiple intelligent agents for execution to obtain execution results; and the planning layer is further configured to feed back the execution results to the target object.

[0012] According to another aspect of the embodiments of this disclosure, an intelligent agent communication and interaction system is also provided, including: a client, configured to send multimodal information, wherein the multimodal information is determined based on the multimodal interaction mode of the target object; a server, connected to the client, configured to generate a trigger signal based on the multimodal information, determine multimodal conversation content using the trigger signal and pre-stored historical dialogue data, perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed, and assign the sequence of actions to be executed to multiple intelligent agents for execution to obtain an execution result; the client is also configured to output the execution result.

[0013] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of the present disclosure during runtime.

[0014] According to another aspect of the embodiments of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to execute any of the above-described intelligent agent communication interaction methods.

[0015] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described intelligent agent communication and interaction methods.

[0016] In this embodiment, a trigger signal is generated based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined by using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. Then, the sequence of actions to be executed is assigned to multiple agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, more efficient and more personalized multimodal interaction for the needs of multiple scenarios. This solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario needs and personalized needs.

[0017] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of the operation of a multimodal multi-agent interaction framework according to an embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0023] Figure 5 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0024] Figure 6 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0025] Figure 7 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of the present disclosure;

[0027] Figure 9 is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of the present disclosure;

[0028] Figure 10 is a schematic diagram of the structure of an intelligent agent communication and interaction device according to an embodiment of the present disclosure;

[0029] Figure 11 is a schematic diagram of another intelligent agent communication interaction device according to an embodiment of the present disclosure;

[0030] Figure 12 is a schematic diagram of another intelligent agent communication interaction device according to an embodiment of the present disclosure;

[0031] Figure 13 is a schematic diagram of another intelligent agent communication interaction device according to an embodiment of the present disclosure;

[0032] Figure 14 is a schematic diagram of another intelligent agent communication and interaction device according to an embodiment of the present disclosure;

[0033] Figure 15 is a structural block diagram of a computing device according to an embodiment of the present disclosure;

[0034] Figure 16 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0038] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the explanation is based on the example of agent communication interaction using a large model in a multimodal interaction exploration scenario.

[0039] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0040] Multilevel memory: The memory management model of a dialogue system, including scene memory, dialogue memory, and long-term memory.

[0041] Multi-modal: By integrating multiple information types such as text, images, voice, and video, it enables more natural and comprehensive information interaction and understanding.

[0042] Multi-Agent: A collaborative system composed of multiple autonomous intelligent agents that improves overall efficiency through distributed decision-making, task allocation, and cooperation.

[0043] Perception layer: The basic module for the system to receive external information, responsible for data acquisition, preprocessing and feature extraction (such as visual recognition and speech-to-text).

[0044] Action planning: Generates a sequence of actions for the agent based on the environmental state and user instructions.

[0045] Interactive system: A framework that supports two-way information exchange between humans and machines or intelligent agents, including input parsing, feedback generation, and execution feedback closed loop.

[0046] Enhanced multimodal search: Integrating cross-modal semantic matching technology (such as text-image joint retrieval) to improve the accuracy and speed of information retrieval in complex scenarios.

[0047] Voice Activity Detection (VAD): A technique for identifying speech and non-speech segments in an audio signal, used for noise reduction, speech endpoint detection, and resource optimization.

[0048] Large-scale models need to understand the world's knowledge to approach Artificial General Intelligence (AGI). This requires breaking through the boundaries of pure text (text-only) models. In recent years, a large amount of work has emerged attempting to jointly pre-train text with other modalities to achieve multimodal understanding. Simultaneously, advancements in large-scale and multimodal models have also brought about a new wave of AI hardware applications, with manufacturers of smart cockpits and smart speakers releasing large-scale models and upgrading their performance.

[0049] Currently, most multimodal interaction systems adopt end-to-end speech modeling solutions. These solutions lack tooling capabilities and have limited customizability, making them difficult to scale realistically to business (B2B) customer scenarios. They are more often used as toys for casual conversation and companionship. Furthermore, these solutions lack long-term memory and user personalization capabilities. An end-to-end full-duplex speech modeling solution has also been proposed, where the model directly receives streaming speech input and generates streaming speech output. This solution is non-interventional, does not support video input, and faces serious issues with language model degradation. In addition, traditional speech interaction frameworks are rule-based and lack generalization capabilities, failing to meet the performance requirements of end users.

[0050] The multimodal interaction systems of related technologies have the following drawbacks.

[0051] Defect 1: The end-to-end speech modeling solution fails to effectively integrate external tools or knowledge bases, limiting its functional scalability and practicality in real-world scenarios. Furthermore, the model has low customizability, making it difficult to adapt to the diverse needs of B2B clients and extending to more business scenarios. In addition, the lack of an effective long-term memory mechanism means that user preferences and historical information cannot be personalized during interactions, reducing the continuity and satisfaction of the user experience.

[0052] Defect 2: The end-to-end duplex speech modeling solution only uses duplex speech modeling and does not support video input, which limits the comprehensiveness of scene understanding and information interaction, and faces the problem of insufficient language model understanding ability.

[0053] Defect 3: Interaction frameworks that rely on traditional rules have limited generalization ability when dealing with unknown or complex scenarios, making it difficult to meet the growing intelligent needs of end users.

[0054] To address the aforementioned deficiencies, no effective solution has been proposed prior to this disclosure.

[0055] According to embodiments of this disclosure, an intelligent agent communication interaction method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), Internet, or other types of data networks. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0057] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device acquires the currently input multimodal information and sends the multimodal information to the server; the server generates a trigger signal based on the currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data; the multimodal conversation content is analyzed to obtain a sequence of actions to be executed; the sequence of actions to be executed is assigned to multiple agents for execution to obtain execution results, and the execution results are fed back to the client device (i.e., the client device used by the target object, or the entity device for multimodal interaction). It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of the large model, this embodiment can be performed on the client device.

[0058] It should be noted that with the rapid development of high-performance computing units, the methods provided in this disclosure can also be applied to integrated model machines in other application scenarios. In one optional embodiment, the integrated model machine has multiple built-in models. Users can select one model to adjust as needed to obtain their own model. The high-performance computing unit built into the integrated model machine can then directly call the adjusted model to execute the methods provided in this disclosure. In another optional embodiment, the large integrated model machine has a pre-trained model built-in. Therefore, the high-performance computing unit built into the integrated model machine can directly call this model to execute the methods provided in this disclosure.

[0059] Furthermore, when users need to train their own models, they can upload their own datasets via the client. These datasets are then sent to the server, allowing the server to adjust the pre-trained model using the dataset to obtain the user's customized model, which can then be deployed to the production environment. To facilitate users' model adjustment needs, the server provides complete adjustment tools, development frameworks, and processes, supporting multiple adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0060] Under the above operating environment, this disclosure provides an intelligent agent communication and interaction method as shown in Figure 2. Figure 2 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this disclosure. As shown in Figure 2, the method may include the following steps:

[0061] Step S21: Generate a trigger signal based on the currently input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object;

[0062] Step S22: Determine the multimodal conversation content using the trigger signal and pre-stored historical dialogue data;

[0063] Step S23: Perform conversation analysis on the multimodal conversation content to obtain the sequence of actions to be executed;

[0064] Step S24: Distribute the sequence of actions to be executed to multiple agents for execution, and obtain the execution results;

[0065] Step S25: Feedback the execution result to the target object.

[0066] In this embodiment of the disclosure, the currently input multimodal information can be understood as the multimodal information input by the target object. The target object can be understood as a user or entity that interacts with the smart device in a multimodal manner. It is understood that the smart device may be equipped with the multimodal interaction system proposed in this disclosure. For example, the target object can be a person using a client device, such as a person using smart hardware such as a smartphone, smart wearable device, or smart cockpit, or any object that can communicate with the smart device in multiple ways such as voice, image, and text. For example, sensors in home automation devices (such as temperature and light sensors) can also be part of the target object, sending multimodal data (such as voice broadcast of temperature values ​​or image display of light intensity) to the smart device. There are no limitations here.

[0067] Multimodal information refers to information that simultaneously includes multiple types, such as text, voice, and image information, without limitation. Multimodal information is determined based on the multimodal interaction methods of the target object. Multimodal interaction methods can be understood as combinations of various information types used by the target object to communicate with the smart device, such as interactions combining voice, text, images, video, or gestures, without limitation.

[0068] A trigger signal can be understood as an indication signal generated based on the analysis of multimodal information of the current input, enabling the smart device to respond promptly to the interactive behavior of the target object. In essence, a trigger signal represents the multimodal information of the current input. For example, a smart device can monitor and parse user input information in real time to generate a trigger signal. For instance, when meaningful user activity is detected, such as when the user begins to speak, the environment changes, or the user inputs information through the screen, the smart device generates a corresponding trigger signal.

[0069] In this disclosure, generating a trigger signal based on the current input multimodal information can be understood as follows: the intelligent device first receives the current input multimodal information from the target object, and then generates a specific trigger signal based on the multimodal information. It can be seen that by integrating and analyzing multimodal information, the intelligent device can more accurately understand the user's intent, improve the accuracy of trigger events, and thus provide more intelligent and responsive services.

[0070] After generating a trigger signal, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Historical dialogue data can be understood as pre-stored records of interactions between the user and the smart device, used to build long-term memory and personalized experiences. For example, historical dialogue data includes, but is not limited to, text chat logs, voice dialogue transcripts, image feedback history, video communication information, and user behavior habits. It can be short-term or long-term historical dialogue data; no limitation is imposed here.

[0071] Multimodal conversation content can include multimodal information and the current conversation context determined based on historical dialogue data. It can also include user device information so that the multimodal interaction system can fully understand the user's needs and context, thereby providing more accurate and natural responses.

[0072] In this disclosure, the multimodal conversation content is determined using trigger signals and pre-stored historical dialogue data. This can be understood as the smart device, based on a generated trigger signal, calling upon and integrating the user's historical dialogue data to generate multimodal conversation content. This allows the multimodal interaction system to more comprehensively understand the background and context of the current interaction. Consequently, the interaction becomes more natural and fluid, enhancing user trust and satisfaction with the multimodal interaction system.

[0073] After determining the multimodal conversation content, conversation analysis is performed on the content to obtain a sequence of actions to be executed. This sequence can be understood as a series of operation instructions, or an action list, used to respond to user requests. Specifically, it can be understood as including at least one operation instruction for an action to be executed. For example, semantic understanding, intent recognition, and scenario analysis can be performed on the multimodal conversation content to obtain the sequence of actions to be executed, ensuring that the multimodal interaction system can correctly interpret user needs and formulate corresponding action plans.

[0074] In this disclosure, conversation analysis is performed on multimodal conversation content to obtain a sequence of actions to be executed. This can be understood as the intelligent device performing in-depth analysis of the multimodal conversation content, transforming the user's intent and needs into a specific sequence of actions to be executed. This improves the accuracy and efficiency of the multimodal interaction system in executing tasks, avoids misunderstanding user intent, ensures service precision, and enhances the user experience.

[0075] After obtaining the sequence of actions to be executed, the sequence is distributed to multiple agents for execution, yielding the results. An agent can be understood as a software entity (intelligent agent) with autonomous decision-making and execution capabilities in a multimodal interaction system. Each agent typically focuses on processing a specific modality (such as speech, text, image, video, etc.) or a specific type of task (such as search, translation, visual recognition, etc.). Multiple agents can work independently as well as collaborate to complete complex and diverse user requests or tasks.

[0076] The execution result can be understood as the output generated after multiple intelligent agents complete the task, including response information to user instructions.

[0077] In this disclosure, the sequence of actions to be executed is distributed to multiple intelligent agents for execution, and the execution result is obtained. This can be understood as the intelligent device intelligently assigning tasks from the sequence of actions to the most suitable intelligent agent or multiple intelligent agents for execution. Each intelligent agent handles the corresponding task according to its expertise, and returns the execution result after completing the operation. Therefore, the multimodal interaction system of this disclosure allows multiple intelligent agents to process tasks in parallel, improving processing power and response speed in complex scenarios, while ensuring the accuracy and richness of task execution, thus enhancing the intelligence and flexibility of the multimodal interaction system.

[0078] After obtaining the execution result, the result is fed back to the target object. This can be understood as the smart device sending the execution result back to the target object, thus completing one interaction loop. For example, the smart device can provide feedback through voice replies, image displays, or other forms of output to ensure that the user can understand the system response in a timely and intuitive manner; there are no restrictions on this.

[0079] Understandably, intelligent devices can stream all execution results back to the target object. This means that intelligent devices can continuously and dynamically return the execution results obtained by each agent to the user in the form of a real-time data stream. This ensures the real-time and continuous response of the multimodal interaction system. Users can instantly see or hear the processing results of the multimodal interaction system. The dynamic interaction mode improves the user experience, making the entire conversation smoother and more intuitive. Correspondingly, intelligent devices can also package all execution results into a static, complete output and send it to the user; this is not restricted here.

[0080] As can be seen, this disclosure allows multimodal interaction systems to process multiple types of information simultaneously, thereby gaining a more comprehensive understanding of the user's intent and context, and providing responses that closely resemble natural language communication. Furthermore, by incorporating the user's historical dialogue behavior, the multimodal interaction system can reference personalized information during interaction, enabling smarter and more personalized services, meeting individual needs, and enhancing the interactive intelligence and naturalness of the multimodal interaction system.

[0081] Furthermore, this disclosure ensures the real-time and continuous response of the multimodal interaction system by providing real-time streaming feedback of execution results. Simultaneously, cross-modal multi-agent collaboration is achieved through the division of labor and cooperation among different agents, allowing for the dynamic scheduling of suitable agents to respond to different user scenario needs, thus meeting diverse scenario requirements. For example, switching between voice queries and visual feedback satisfies the diverse and complex interaction needs of users.

[0082] The intelligent agent communication and interaction method provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0083] By employing the embodiments of this disclosure, a trigger signal is generated based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. Then, the sequence of actions to be executed is assigned to multiple intelligent agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This achieves the technical effect of providing more natural, more efficient, and more personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple intelligent agents and cannot meet diverse scenario needs and personalized requirements.

[0084] In an optional embodiment, in step S21, a trigger signal is generated based on the currently input multimodal information, including the following method steps:

[0085] Step S211: Perceive and understand the currently input multimodal information to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through multimodal information;

[0086] Step S212: Generate a trigger signal based on the understanding results.

[0087] It is understood that the smart device is equipped with the multimodal interaction system proposed in this disclosure. The multi-agent interaction framework of the multimodal interaction system may include a perception layer. The perception layer serves as the window through which the entire multimodal interaction system directly interacts with the user. It is used to monitor and parse the user's input information in real time, and then generate trigger signals based on the parsing results.

[0088] In this embodiment of the disclosure, when generating a trigger signal based on the currently input multimodal information, the currently input multimodal information can be perceptually understood to obtain an understanding result, and then the trigger signal can be generated based on the understanding result. The perceptual understanding of the currently input multimodal information can be understood as a processing procedure of the perception layer, used to convert multimodal information into a form understandable by the multimodal interaction system, including recognizing, parsing, and understanding different multimodal information.

[0089] For example, the perception layer can use sound activity detection technology to detect whether the user has started speaking and whether speaking has ended; it can also use visual scene transition detection technology to detect whether the user is moving to a different environment; it can also use image recognition technology to analyze specific objects; and it can use other technologies to analyze text and other information. It is understood that the perception layer can use different detection results to generate trigger signals as needed, and this is not limited here.

[0090] The understanding result can be understood as the information obtained by the perception layer after perceiving and understanding multimodal information. It is used to perceive the intention or action of the target object through multimodal information. That is, the understanding result can contain a preliminary interpretation of the target object's intention or behavior. For example, the understanding result may indicate "the user is asking about the weather" or "the user picked up a book," without limitation.

[0091] In this disclosure, the multimodal information input is perceived and understood. The resulting understanding can be understood as follows: the perception layer in the intelligent device first receives multimodal information from the target object, then analyzes and understands this information, and finally integrates it into an understanding result. This understanding result reflects the target object's intention or ongoing action. Therefore, by perceiving and understanding multimodal information, multimodal interaction can more comprehensively capture user needs, improve the accuracy and response speed of the interaction, and provide precise contextual information for subsequent intelligent processing.

[0092] The perception layer then generates one or more trigger signals based on the obtained understanding results. For example, if the understanding results indicate that the user is asking about the weather, the perception layer will generate a trigger signal related to the weather query. If the user is describing a visual scene, the perception layer will generate a trigger signal related to image recognition or visual search, and so on.

[0093] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include a control layer, which receives trigger signals from the perception layer. This can be understood as the perception layer sending the generated trigger signals to the control layer, which then instructs the control layer on how to handle subsequent interactions.

[0094] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0095] Step S201: Obtain historical dialogue data from the dialogue memory storage area, wherein the dialogue memory storage area is used to maintain the historical information of multiple rounds of dialogue between the target object within a preset time range.

[0096] In this embodiment, the dialogue memory storage area is used to maintain the multi-turn dialogue history information of the target object within a preset time range. It is understood that this disclosure proposes a multi-level memory architecture, and the dialogue memory storage area is a component of this architecture, namely, dialogue memory, used to store multi-turn dialogue history information related to a specific scenario or environment. It is also understood that the dialogue memory storage area's maintenance of the target object's multi-turn dialogue history information within the preset time range involves not only storing the multi-turn dialogue history information but also a series of management and processing methods for it, such as dynamic updating, efficient retrieval, and in-depth understanding and utilization of the information. These are not limited here.

[0097] Multi-turn dialogue history can be understood as the history of a user's interactions with a multimodal interaction system on a certain hardware device (such as smart glasses or smartwatches), or a record of interactions conducted by the user in a specific location (such as home, office, or coffee shop). There are no restrictions on this.

[0098] The preset time range can be a pre-defined time window, usually a recent period of time (such as the past few hours, day, or week), thereby maintaining the timeliness and relevance of scene memory.

[0099] In this embodiment of the disclosure, retrieving historical dialogue data from the dialogue memory storage area can be understood as the intelligent device accessing the dialogue memory storage area, determining which part of the dialogue memory needs to be called based on the current interaction scenario, and then retrieving the relevant historical dialogue data from it. The historical dialogue data will be used in subsequent decision-making processes to generate responses that are more tailored to the user's scenario and needs. Exemplarily, the control layer in a multi-agent interaction framework can retrieve historical dialogue data from the dialogue scenario memory storage area; this is not limited to this.

[0100] It can be seen that by using dialogue memory, multimodal interaction systems can provide more contextualized services based on the user's current environment and recent interaction history. At the same time, by utilizing historical dialogue data, multimodal interaction systems can establish the contextual chain of user dialogues, ensuring that responses to users are not only accurate but also coherent, avoiding problems such as "fragmentation" or "contextual asynchrony".

[0101] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0102] Step S202: In response to receiving authorization from the target object, the historical dialogue data is preprocessed to obtain a processing result, wherein the processing result is used to describe the target object's preference content;

[0103] Step S203: Store the processing result in the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through active recording or passive recording. Active recording records the target object's preference content according to the target object's instructions. Passive recording uses an information extraction model to extract the target object's preference content from the target object's dialogue history information.

[0104] It is understood that the multi-level memory architecture proposed in this embodiment may further include a long-term memory storage area, i.e., long-term memory. The long-term memory storage area is used to store important information such as user preference content for a long period, and to perform a series of management and processing on the user preference content. Furthermore, the long-term memory storage area is used to maintain the preference content of the target object from the perspective of the target object, through active recording or passive recording.

[0105] The active recording method records the target user's preferences based on the target user's instructions. This can be understood as the user directly informing the multimodal interaction system of their preferences that they want to be remembered, and the multimodal interaction system records them accordingly. For example, a user can give the instruction "Remember this document I'm looking at," and the multimodal interaction system will then actively store the document in long-term memory.

[0106] The passive recording method uses an information extraction model to extract the target object's preferences from the target object's dialogue history. This can be understood as a multimodal interaction system using an information extraction model to analyze the user's dialogue history in multiple situations. The information extraction model automatically identifies and learns the user's preferences and stores them in long-term memory without explicit user instruction.

[0107] Authorization of the target user can be understood as the user granting the multimodal interaction system permission to process their personal information (including historical conversation data), which usually follows privacy policies and data protection regulations.

[0108] Preprocessing can include cleaning, format conversion, and feature extraction of historical dialogue data to make it suitable for subsequent processing; this is not limited. Preprocessing can also include text segmentation, speech transcription, image recognition, and using natural language processing techniques and other data analysis methods to understand the meaning of user expressions. The preprocessed results are used to describe information about user preferences for subsequent personalized interactions; these results may include, for example, favorite music genres and frequently visited restaurants; this is not limited.

[0109] In this embodiment of the disclosure, the acquired historical dialogue data can be refined and filtered before being stored in a long-term memory storage area. For example, the control layer in the multi-agent interaction framework first performs a series of preprocessing operations on the historical dialogue data to extract user preference content and obtain a processing result describing the user's preferences. Then, the control layer stores the processing result in the long-term memory storage area, ensuring the long-term preservation of user preference information. This allows the multimodal interaction system to continuously respond according to user preferences even during interactions across devices and time periods.

[0110] As can be seen, the long-term memory storage area allows user preferences to be shared across different scenarios and devices, ensuring that the personalized services provided by the multimodal interaction system remain consistent in any situation. Furthermore, as users spend more time using the system, the preferences stored in long-term memory are continuously enriched and updated, thereby enabling continuous optimization and evolution of the service.

[0111] In an optional embodiment, in step S23, session analysis is performed on the multimodal session content to obtain a sequence of actions to be executed, including the following method steps:

[0112] Step S231: Identify the target object's interaction intent from the multimodal conversation content using a semantic rule engine;

[0113] Step S232: In a known scenario, control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, control the zero-sample action planning model to generate a sequence of actions to be executed based on the interaction intent.

[0114] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include a planning layer. The planning layer is used to receive and analyze the multimodal conversation content sent by the control layer, break down the user's instructions into a list of actions to be executed and attach corresponding parameters.

[0115] In this embodiment of the disclosure, when performing conversation analysis on multimodal conversation content to obtain a sequence of actions to be executed, a semantic rule engine can be used to identify the interaction intent of the target object from the multimodal conversation content. The semantic rule engine can be a tool based on predefined grammatical rules and semantic models, used to parse input multimodal data and understand user intent. The semantic rule engine can process and recognize different forms of information such as text, speech, and images, and transform user input into executable commands or requests through pattern matching and semantic analysis. For example, the semantic rule engine can be a traditional semantic understanding engine based on Lightweight Grammar Formalism (LGF) grammars, matching user intent through exhaustive methods. The semantic rule engine can also parse user instructions using Speech Act Theory to identify the specific speech act type behind them, thereby responding to user intent more accurately; however, this is not limited to specific examples.

[0116] Interaction intent can be understood as the action goals, needs, or requests expressed by users through multimodal conversation content.

[0117] The fine-tuned action planning model can be understood as a large action planning model that has undergone fine-tuning (FT). It's used to collect data for user command scenarios and then fine-tunes the large model to obtain a list of actions. Fine-tuning is a common step in machine learning, especially deep learning. It involves further training a pre-trained model using a small amount of data specific to a particular task, allowing the model to better adapt to the characteristics of that task or specific dataset. In essence, it means that under a known scenario, the pre-trained action planning model is specifically fine-tuned to better adapt to the interpretation and response planning of user commands in a particular scenario, thus obtaining the fine-tuned action planning model.

[0118] Zero-shot action planning models can generate action items using a strong base model and text prompts. This can be understood as follows: in unknown or new scenarios, zero-shot action planning models can generate a reasonable sequence of actions to be executed based on the user's interaction intent, even with limited or no specific scenario data.

[0119] In this disclosure, identifying the target object's interaction intent from multimodal conversation content using a semantic rule engine can be understood as the planning layer in the multi-agent interaction framework using the semantic rule engine to deeply analyze the multimodal conversation content and identify the user's specific interaction intent. For example, if a user says, "I want to know tomorrow's weather," and simultaneously displays a screenshot of a weather forecast, the semantic rule engine will combine the speech recognition and image recognition results to determine that the user wants to know about tomorrow's weather conditions; this is not a limitation. Therefore, multimodal information can be accurately parsed, ensuring that the multimodal interaction system accurately understands the user's intent and avoids misunderstandings or improper handling.

[0120] Subsequently, the planning layer can control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent in a known scenario, or control the zero-shot action planning model to generate a sequence of actions to be executed based on the interaction intent in an unknown scenario. In other words, the planning layer will choose to use either the fine-tuned action planning model or the zero-shot action planning model to generate the sequence of actions to be executed based on the identified interaction intent.

[0121] Specifically, in known scenarios, such as when users frequently inquire about the weather at home, the planning layer uses an action planning model fine-tuned based on the data from that scenario to generate a response, thereby enabling a more accurate and rapid response.

[0122] In unknown scenarios, such as when a user makes a new request that the multimodal interaction system has never encountered before, the planning layer will use a zero-shot action planning model. This zero-shot action planning model can handle new scenarios more flexibly and generate action sequences based on a wider range of data and knowledge.

[0123] It can be seen that by dynamically selecting the fine-tuned action planning model and the zero-shot action planning model, the multimodal interaction system can generate reasonable responses in both known and unknown scenarios, thereby improving scenario adaptability and generalization ability.

[0124] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0125] Step S204: Based on the interaction intent, determine whether to use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through active recording or passive recording. Active recording records the target object's preference content according to the target object's instructions. Passive recording uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragment is used to generate personalized responses in conjunction with the action sequence to be executed.

[0126] Understandably, the planning layer also possesses tool invocation and memory activation discrimination functions. For example, the planning layer can use a text discrimination model to determine whether the current user command needs to enable search enhancement and whether the current user command needs to be associated with the user's past memories. If the determination is "needed," a multimodal search can be used to search for relevant content in the long-term memory storage area and send the relevant content to the downstream execution agent, which can then generate a response that matches the user's historical preferences.

[0127] In this embodiment of the disclosure, the multimodal search method can be understood as a method of information retrieval that combines multiple information types such as text, voice, images, and video, aiming to understand user needs more comprehensively and accurately and provide personalized services.

[0128] Memory fragments can be understood as historical records or preference information related to user interaction intentions stored in the long-term memory storage area. Memory fragments are used to generate personalized responses in conjunction with the sequence of actions to be performed, that is, they can be used as a reference when generating personalized responses.

[0129] Personalized responses can be understood as responses or suggestions that are highly matched to the user's specific needs, generated based on the user's preferences and historical behavior patterns.

[0130] In this disclosure, determining whether to retrieve memory fragments of the interaction intent from long-term memory storage using a multimodal search method based on the interaction intent can be understood as the planning layer using a decision mechanism to decide whether to retrieve relevant information from long-term memory storage based on the user's expressed interaction intent. If the decision result is "needed," a multimodal search method will be used to find memory fragments related to the user's current request. The found memory fragments will be combined with the generated sequence of actions to be executed, thereby generating a response that is more tailored to the user's personality.

[0131] For example, if a user once said "I love hot pot", this content will be stored in the long-term memory storage area. A few days later, when the user says "What are some good places to eat nearby", the system will use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The multimodal interaction system will then prioritize recommending hot pot-related items and highlight them in the reply.

[0132] It can be seen that by determining whether to recall memory fragments from long-term memory storage and combining this with enhanced multimodal search, the multimodal interaction system can provide more personalized and context-relevant responses, i.e., highly personalized services. This not only improves the user experience but also demonstrates the intelligence and flexibility of the multimodal interaction system in understanding user needs. Furthermore, the execution result is guided and influenced by the personalized response, meaning that the execution result can be generated based on user preferences and historical information, ensuring the targetedness and effectiveness of the action.

[0133] In an optional embodiment, in step S24, the sequence of actions to be executed is assigned to multiple agents for execution to obtain the execution result, including the following method steps:

[0134] Step S241: Distribute the action sequence to be executed to modally matched agents among multiple agents for independent execution, and / or distribute the action sequence to cross-modal agents among multiple agents for cooperative execution, and obtain the execution result.

[0135] It is understandable that the multi-agent interaction framework of a multimodal interaction system can also include an execution layer. For example, the execution layer can consist of a router, a large number of agents with various modalities, and dialogue functions. The router receives the action list output by the planning layer, i.e., the sequence of actions to be executed, and distributes the sequence of actions to different agents for execution. It is important to note that after an agent completes its execution, it returns the execution result to the planning layer. The planning layer then determines whether all the sequences of actions to be executed have been completed. After all the sequences of actions to be executed have been completed, all execution results are streamed back to the user.

[0136] In this embodiment of the disclosure, when the sequence of actions to be executed is assigned to multiple agents for execution and the execution result is obtained, the sequence of actions to be executed can be assigned to agents with matching modalities among the multiple agents for independent execution. This can be understood as the execution layer assigning each instruction in the sequence of actions to be executed to an agent that specializes in processing data of a specific modality. For example, text instructions may be sent to a text processing agent, and voice instructions may be sent to a voice processing agent; this is not limited here. Therefore, agents with matching modalities can quickly and accurately process data corresponding to their own modality, reducing latency and errors in cross-modal processing, while providing higher quality execution results.

[0137] Understandably, after independent execution, the execution results from different agents will be integrated. For example, based on task requirements and user preferences, a comprehensive judgment will be made to select or merge the most relevant results to form a unified feedback strategy.

[0138] And / or, the sequence of actions to be executed is distributed to cross-modal agents among multiple agents for collaborative execution to obtain the execution result. This can be understood as follows: for tasks requiring the combination of multiple modal data, the execution layer assigns the sequence of actions to agents capable of cross-modal collaboration. These agents share information and work together to complete complex tasks. Thus, cross-modal agents can combine multiple information types to provide more comprehensive and accurate understanding and execution capabilities. For example, when processing a request involving both voice and images, they can provide better suggestions through comprehensive analysis. Simultaneously, through cooperation among multiple agents, complex tasks involving multiple modalities can be executed, improving the functionality and user satisfaction of multimodal interaction systems.

[0139] For example, based on the modal characteristics involved in a series of action instructions to be executed, such as whether they include speech recognition, image analysis, or text understanding, these instructions can be intelligently assigned to appropriate cross-modal agents. Each agent then independently and efficiently completes the task within its specialized modal domain. During this process, agents can share necessary information, such as keywords, contextual understanding, or user preferences, through internal system communication mechanisms, thereby enhancing each other's processing capabilities and decision-making quality. Once all agents have completed their assigned tasks, the execution results from these different agents are integrated to form a unified, coherent, and insightful feedback, which is ultimately presented to the user, ensuring that the interaction process is both rich and diverse, and closely aligned with the user's specific needs and context.

[0140] For example, when both independent and collaborative execution exist, action instructions involving different modalities can be precisely assigned. For instance, voice instructions are assigned to a voice agent, and image tasks to a vision agent, ensuring that each agent can independently and in parallel process its specialized modality task. Simultaneously, through coordination, cross-modal agents can share necessary information, conduct collaborative execution, and ultimately integrate the results of all independent and collaborative executions to provide the user with a comprehensive and coherent response.

[0141] As can be seen, this disclosure considers that user-side input may include various information types such as text, audio, and visual information. Therefore, a cross-modal multi-agent scheme is designed at the execution layer, enabling agents of different modalities to autonomously complete tasks they are good at, or to interact and cooperate to complete tasks. Through the combined use of independent agents and cross-modal agents, the multimodal interaction system can efficiently and accurately process user requests, providing a personalized and high-quality service experience. It also enables the multimodal interaction system to handle a wider range of tasks, demonstrating the flexibility and scalability of the multimodal interaction system's intelligence. Furthermore, through the specialization and collaboration of agents, the multimodal interaction system can more effectively allocate resources, reduce redundant calculations, and improve the speed and efficiency of task processing.

[0142] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0143] Step S205: Store the execution result in the scene memory storage area, whereby the scene memory storage area is used to temporarily store the execution results of different actions of multiple intelligent agents.

[0144] It is understood that the multi-level memory architecture proposed in this embodiment may further include a scene memory storage area, i.e., scene memory. The scene memory storage area is used to temporarily store the execution results of agents related to the current interaction scene, that is, to temporarily store the execution results of different actions of multiple agents. It can be understood as a form of short-term memory used to support ongoing dialogues or tasks.

[0145] In this embodiment of the disclosure, storing the execution result in the scene memory storage area can be understood as the execution layer storing the execution result in the scene memory storage area. For example, if a user requests through smart glasses, "Teach me how to make this dish," the multimodal interaction system may invoke a voice assistant agent for dialogue understanding and guidance, while simultaneously invoking a visual recognition agent to analyze the recipe image. After the agents complete their respective tasks, the execution results are sent back to the scene memory storage area, which may include "the ingredients needed for this dish," "voice guidance on the preparation steps," and "key information from the recipe image," etc., without limitation.

[0146] Therefore, the scene memory storage area can provide temporary data caching for the current interaction scene, ensuring that all relevant action results are recorded during task execution or within the interaction loop for later use or reference. This helps maintain the contextual coherence of the dialogue or task and avoids information loss. At the same time, the information stored in the scene memory can be accessed and integrated by subsequent agents or planning layers, helping the multimodal interaction system make more comprehensive and accurate decisions.

[0147] In an optional embodiment, step S25 involves feeding back the execution result to the target object, including the following method steps:

[0148] Step S251: Determine whether all actions in the action sequence to be executed have been completed;

[0149] In step S252, in response to the completion of all multiple actions, the execution results are fed back to the target object using a streaming method.

[0150] In this embodiment of the disclosure, streaming transmission is a data transmission method that can continuously send data and is suitable for scenarios that require real-time interaction and feedback.

[0151] In this disclosure, when feeding back the execution results to the target object, it can first be determined whether all actions in the sequence of actions to be executed have been completed. In response to the completion of all actions, the execution results are fed back to the target object using a streaming method. This can be understood as the multimodal interaction system confirming whether all agents have completed their assigned tasks; that is, the multimodal interaction system checks whether each action in the sequence of actions to be executed has been correctly executed and completed by the agent to ensure that all execution results are collected. If the multimodal interaction system confirms that all agents have completed their tasks, it will use a streaming method to feed back the execution results to the target object in real time to ensure the timeliness and continuity of information transmission.

[0152] Therefore, streaming transmission ensures timely feedback of execution results, reducing user waiting time. This is especially beneficial for scenarios requiring immediate responses, such as online shopping inquiries and navigation guidance, greatly enhancing the smoothness and satisfaction of the user experience. In other words, this disclosed multimodal interaction system not only ensures that all user-requested operations are fully executed, but also presents the execution results to the user in the smoothest and most timely manner.

[0153] In an optional embodiment, the intelligent agent communication interaction method further includes the following method steps:

[0154] Step S206: In response to meeting preset conditions, the number of agents and / or the multimodal processing capabilities of the multiple agents are expanded.

[0155] In this embodiment of the disclosure, in response to the satisfaction of preset conditions, the number of agents and / or the multimodal processing capabilities of multiple agents are expanded. This can be understood as the multimodal interaction system continuously evaluating the current operating status and user needs. Once the preset conditions are detected, the multimodal interaction will automatically expand the number of agents and / or the multimodal processing capabilities of multiple agents.

[0156] Among them, meeting the preset conditions may include increased complexity of user requests (e.g., input content includes multiple types of information), high current load pressure of the multimodal interaction system, and high resource utilization (e.g., the utilization rate of computing power and storage space is close to the preset upper limit), etc., which are not restricted here.

[0157] Through the aforementioned extension mechanism, it is ensured that the multimodal interaction system can dynamically adjust resources according to actual needs, so as to handle increasingly complex or high-concurrency interaction tasks more efficiently.

[0158] For example, in a smart cockpit scenario, if the multimodal interaction system detects multiple passengers simultaneously issuing different interaction requests (such as adjusting seats, checking traffic conditions, or playing music), and these requests involve data from multiple modalities, the system will decide whether to increase the number of agents or enhance the multimodal processing capabilities of a single agent based on preset load management and resource optimization strategies. Thus, by increasing the number of agents, the multimodal interaction system can handle more user requests simultaneously, supporting stable operation in high-concurrency scenarios and avoiding response delays or task failures due to insufficient processing power. Simultaneously, by expanding multimodal processing capabilities, the system can more flexibly respond to complex or changing interaction scenarios, supporting more diverse user needs and interaction methods, thereby improving the service quality and user experience of the multimodal interaction system.

[0159] It can be seen that the multi-agent interaction framework disclosed herein is forward-looking, taking into account the scalability of the multi-modal agent, enabling the multi-modal agent to adapt to more complex and broader multi-modal interaction needs when necessary by adding new functions, improving processing capabilities, or optimizing algorithms.

[0160] Figure 3 is a schematic diagram of the operation of a multimodal multi-agent interaction framework according to an embodiment of the present disclosure. As shown in Figure 3, the external environment includes the user, the user's environmental information, and dynamic knowledge (such as real-time weather, traffic conditions, etc.).

[0161] The multi-agent interaction framework can interact with users in real time across all modes using different hardware devices (such as smart cockpits, smart wearables, mobile phones, and smart screens as shown in Figure 3). Specifically, the multi-agent interaction framework can include four layers: perception layer, control layer, planning layer, and execution layer.

[0162] The perception layer, serving as the direct window for interaction between the entire multimodal interaction system and the user, receives multimodal information input from the device in real time, performs perception and understanding of each modality, and generates trigger signals based on the understanding results, sending these signals along with the modal content to the control layer. For example, the perception layer may employ sound activity detection technology to identify sound events (such as when a user begins or stops speaking), but it can also use speech recognition technology, video perception technology, etc., for perception and understanding; this is not a limitation. It is understood that the multimodal interaction system can use different technologies for perception and understanding to generate trigger signals as needed.

[0163] The control layer receives trigger signals from the perception layer and manages the history of short-term conversations. Combining user device information and historical records, it organizes the information into multimodal content and sends it to the planning layer. On the other hand, with authorization, the control layer also refines, filters, and stores the user's conversation history information in the long-term memory module.

[0164] The planning layer receives and analyzes the multimodal conversation content input from the control layer, breaking down user commands into sequences of actions to be executed and attaching corresponding parameters. Specifically, the planning layer includes the following modules: a semantic rule engine, a fine-tuned action planning model, a zero-shot action planning model, and tool invocation and memory activation discrimination. For detailed descriptions, please refer to the aforementioned sections; they will not be repeated here.

[0165] The execution layer consists of a router and a large number of agents with various modalities and dialogue functions. The router receives the sequence of actions to be executed from the planning layer and assigns it to different agents (such as the voice assistant agent, visual question-answering agent, bartending / chess teaching agent, voice translation agent, interactive podcast agent, etc. in Figure 3) for execution. After each agent completes its execution, it returns the result to the planning layer. The planning layer determines whether all actions have been completed and, if so, streams all execution results back to the user.

[0166] Based on the aforementioned multi-agent interaction framework, this disclosure achieves the following three important characteristics through the addition of some key components:

[0167] The first type is a multi-level memory storage system.

[0168] This disclosure presents a multi-level memory architecture comprising scene memory, dialogue memory, and long-term memory for recording user logs and preference information. Scene memory is used for temporary storage of single action executions, dialogue memory is used to maintain the user's multi-turn dialogue history within a short period, and long-term memory records user preferences through active or passive methods, with the user as the central dimension.

[0169] The second type is personalized interaction.

[0170] With the support of a multi-level memory architecture, multimodal interaction systems can become increasingly intelligent during interaction. Specifically, this disclosure uses a tool call and memory activation discrimination module at the planning layer to determine whether the user's current command requires memory enhancement. If the determination is "yes," a multimodal search is used to search for relevant content in the long-term memory storage area, and this content is sent to the downstream execution agent. The agent can then generate a response that matches the user's historical preferences.

[0171] The third type is cross-modal multi-agent systems.

[0172] Considering that user-side input may include multimodal information such as text, audio, and visual data, this disclosure designs a cross-modal multi-agent scheme at the execution layer, enabling different modal agents to autonomously complete tasks they are good at, or to interact and cooperate to complete tasks. On the other hand, the multi-agent interaction framework of this disclosure also considers the scalability of the full-modal agent, allowing it to adapt to more complex and broader multimodal interaction needs when necessary by adding new functions, improving processing power, or optimizing algorithms.

[0173] For example, the above steps can be implemented using a full-modal understanding and generative model combined with a long-term memory multimodal search enhancement framework, without limitation here.

[0174] It is easy to understand that the beneficial effects of the intelligent agent communication and interaction method provided in this disclosure include the following points.

[0175] Beneficial Effect (1): This disclosure proposes a multi-level memory structure that generates a user-personalized multimodal knowledge base by accepting the user's active memory instructions and passively mining and analyzing the user's dialogue history (under authorization). When the user interacts with the multimodal interaction system, the planning layer will make a decision on whether to activate long-term memory based on the current user instructions and multimodal information, and recall the corresponding memory fragments to generate a personalized response.

[0176] Beneficial effect (2): This disclosure realizes multimodal interaction by splitting multiple modules, giving full play to the potential of various single-modal better models and intelligent agents, and ensuring intervention and business scalability.

[0177] Beneficial effect (3): This disclosure designs a cross-modal multi-agent structure, which supports the interaction of multiple agents while preserving the better interaction effect of different modalities as much as possible. For example, in the voice assistant scenario, when the user says "teach me to play chess", the video chess teaching agent will be invoked to interact with the user.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0179] Furthermore, it should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this disclosure.

[0181] According to embodiments of this disclosure, a method for intelligent agent communication and interaction, as shown in FIG4, is also provided. FIG4 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this disclosure. As shown in FIG4, the method includes:

[0182] Step S41: Generate a trigger signal based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0183] Step S42: Determine the multimodal merchant recommendation conversation content using the trigger signal and pre-stored historical merchant recommendation dialogue data;

[0184] Step S43: Perform conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0185] Step S44: Distribute the merchant recommendation action sequence to multiple agents for execution to obtain the merchant recommendation execution result;

[0186] Step S45: Feedback the merchant recommendation execution results to the target audience.

[0187] In this embodiment of the disclosure, a multimodal merchant recommendation request can be understood as a user expressing a need for merchant recommendations to the multimodal interaction system through multiple modalities (such as voice, text, and images). For example, saying "What are some good restaurants nearby?" via voice is not limited here.

[0188] Historical merchant recommendation dialogue data can be understood as the dialogue records and results of a user's past merchant recommendation requests stored in the long-term memory of a multimodal interaction system, including information such as user preferences, historical feedback, and choices.

[0189] The merchant recommendation action sequence can be understood as a series of action instructions generated by the multimodal interaction system based on the conversation analysis results. These instructions guide the subsequent agent to perform specific merchant recommendation tasks, such as filtering conditions, sorting rules, and presentation methods.

[0190] The results of merchant recommendation execution can be understood as the specific results such as recommendation lists, map markers, and voice responses obtained by multiple intelligent agents after performing merchant recommendation tasks according to the assigned action sequence.

[0191] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0192] The intelligent agent communication and interaction method provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0193] By employing embodiments of this disclosure, a trigger signal is generated based on the current input multimodal merchant recommendation request. The multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object. Then, the trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the multimodal merchant recommendation session content. Next, session analysis is performed on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence. This sequence is then distributed to multiple agents for execution, yielding merchant recommendation execution results. Finally, the execution results are fed back to the target object. This achieves efficient cross-modal multi-agent collaboration, providing a more natural, efficient, and personalized multimodal interaction for various scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0194] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0195] According to embodiments of this disclosure, a method for intelligent agent communication and interaction, as shown in FIG5, is also provided. FIG5 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this disclosure. As shown in FIG5, the method includes:

[0196] Step S51: Obtain the agent communication interaction request through the first application programming interface. The request data carried in the agent communication interaction request includes: multimodal information, which is determined according to the multimodal interaction mode of the target object.

[0197] Step S52: Return the agent communication interaction response through the second application programming interface. The response data carried in the agent communication interaction response includes: the execution result, which is obtained according to any of the above agent communication interaction methods.

[0198] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0199] In this embodiment of the disclosure, the Application Programming Interface (API) is a software interface that allows different software applications to interact. The API defines how applications communicate, exchange data, and call services.

[0200] A communication interaction request from an intelligent agent can be understood as a request sent from another intelligent agent or application, requesting the multimodal interaction system to perform a specific multimodal interaction task or operation. For example, a request might be sent from the voice assistant intelligent agent of smart glasses, asking the multimodal interaction system to identify the image currently being viewed by the user and provide relevant information.

[0201] Intelligent agent communication interaction response can be understood as the result returned to the requester (intelligent agent or application) through the API interface after the multimodal interaction system has completed the requested task, which includes specific feedback or information on the execution of the task.

[0202] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0203] The intelligent agent communication and interaction method provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0204] By employing the embodiments of this disclosure, an agent communication interaction request is obtained through a first application programming interface (API). The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, an agent communication interaction response is returned through a second API. The response data carried in the agent communication interaction response includes an execution result, which is obtained according to any of the above agent communication interaction methods. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. It also solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0205] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0206] According to embodiments of this disclosure, a method for intelligent agent communication and interaction, as shown in FIG6, is also provided. FIG6 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this disclosure. As shown in FIG6, the method includes:

[0207] Step S61: Obtain the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, and the multimodal information is determined according to the multimodal interaction mode of the target object;

[0208] Step S62: In response to the agent communication interaction dialogue request, return the agent communication interaction dialogue reply, wherein the information carried in the agent communication interaction dialogue reply includes: execution result, the execution result is obtained according to any of the above agent communication interaction methods;

[0209] Step S63: Display the execution results in the graphical user interface.

[0210] In this embodiment of the disclosure, an intelligent agent communication interaction dialogue request can be understood as a request from another intelligent agent or user, requiring the multimodal interaction system to respond to a multimodal dialogue task, such as answering a question, providing advice, or performing an operation. The request data contains multimodal information, that is, it may contain various types of data such as text, voice, and images.

[0211] Intelligent agent communication and interactive dialogue responses can be understood as feedback or answers generated by a multimodal interaction system in response to received multimodal requests. The information carried in the response includes the execution result, that is, the specific output generated by the multimodal interaction system after performing the task according to the request, such as a recommendation list, question answers, operation confirmation, etc.

[0212] A graphical user interface (GUI) is a user interface that uses graphical elements (such as buttons, sliders, menus, and windows) to display information and receive user input. It is widely used in smart devices and applications, providing an intuitive and convenient user operating environment.

[0213] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0214] The intelligent agent communication and interaction method provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0215] By employing the embodiments of this disclosure, a current input agent communication interaction dialogue request is obtained. The request data carried in the agent communication interaction dialogue request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, in response to the agent communication interaction dialogue request, an agent communication interaction dialogue reply is returned. The information carried in the agent communication interaction dialogue reply includes an execution result, which is obtained according to any of the aforementioned agent communication interaction methods. Finally, the execution result is displayed in a graphical user interface. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for needs in multiple scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0216] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0217] According to embodiments of this disclosure, a method for intelligent agent communication and interaction, as shown in FIG7, is also provided. FIG7 is a flowchart of an intelligent agent communication and interaction method according to an embodiment of this disclosure. As shown in FIG7, the method includes:

[0218] Step S71: In response to the input command applied to the operation interface, display multimodal information on the operation interface;

[0219] Step S72: In response to the processing command applied to the operation interface, display the execution result on the operation interface; wherein the execution result is obtained according to any of the above-mentioned agent communication interaction methods.

[0220] In this embodiment of the disclosure, the user interface can be understood as the interface through which a user interacts with a smart device or application, and may be a touch screen, a voice control interface, or other forms of user input interface.

[0221] Input commands can be understood as commands issued by the user through the user interface, used to provide operation or query requests to a multimodal interaction system. In multimodal interaction scenarios, input commands may include various forms of data such as voice, text, and images.

[0222] Processing instructions can be understood as instructions given by users or intelligent agents to the multimodal interaction system to perform specific operations or tasks through the user interface, such as searching for information, playing music, or setting reminders.

[0223] For a detailed description, please refer to the description of the foregoing embodiments, which will not be repeated here.

[0224] The intelligent agent communication and interaction method provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0225] By employing the embodiments of this disclosure, multimodal information is displayed on the operation interface in response to input commands applied to the operation interface, and then, in response to processing commands applied to the operation interface, the execution result is displayed on the operation interface. The execution result is obtained according to any of the aforementioned agent communication and interaction methods, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This realizes the technical effect of providing more natural, efficient, and personalized multimodal interaction for needs in multiple scenarios, and solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0226] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0227] According to an embodiment of this disclosure, an intelligent agent communication and interaction system as shown in FIG8 is also provided. FIG8 is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this disclosure. As shown in FIG8, the system includes:

[0228] The perception layer is used to generate trigger signals based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object;

[0229] The control layer is used to determine the content of a multimodal conversation using trigger signals and pre-stored historical dialogue data;

[0230] The planning layer is used to perform session analysis on the multimodal session content to obtain the sequence of actions to be executed;

[0231] The execution layer is used to distribute the sequence of actions to be executed to multiple agents for execution and to obtain the execution results.

[0232] The planning layer is also used to feed back the execution results to the target object.

[0233] In this embodiment of the disclosure, the intelligent agent communication and interaction system can execute the above-described intelligent agent communication and interaction method, as detailed in the description of the foregoing embodiments, which will not be repeated here.

[0234] The intelligent agent communication and interaction system provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0235] By employing the embodiments of this disclosure, an intelligent agent communication and interaction system is used to achieve the goal of efficient collaboration among cross-modal multi-agents. This enables the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios, thereby solving the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0236] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0237] According to an embodiment of this disclosure, an intelligent agent communication and interaction system as shown in FIG9 is also provided. FIG9 is a schematic diagram of the structure of an intelligent agent communication and interaction system according to an embodiment of this disclosure. As shown in FIG9, the system includes:

[0238] The client is used to send multimodal information, which is determined based on the multimodal interaction method of the target object;

[0239] The server connects to the client and is used to generate trigger signals based on multimodal information, determine the multimodal conversation content using the trigger signals and pre-stored historical dialogue data, perform conversation analysis on the multimodal conversation content to obtain the sequence of actions to be executed, and assign the sequence of actions to be executed to multiple agents for execution to obtain the execution result.

[0240] The client is also used to output the execution results.

[0241] In this embodiment of the disclosure, the intelligent agent communication and interaction system can execute the above-described intelligent agent communication and interaction method, as detailed in the description of the foregoing embodiments, which will not be repeated here.

[0242] The intelligent agent communication and interaction system provided in this disclosure can be applied, but is not limited to, to multimodal interaction application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, multimodal interaction scenarios related to e-commerce services, multimodal interaction scenarios related to education services, multimodal interaction scenarios related to legal services, and multimodal interaction scenarios related to medical services are not limited here.

[0243] By employing the embodiments of this disclosure, an intelligent agent communication and interaction system is used to achieve the goal of efficient collaboration among cross-modal multi-agents. This enables the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios, thereby solving the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0244] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0245] According to embodiments of this disclosure, an apparatus embodiment for implementing the above-described intelligent agent communication interaction method is also provided. Figure 10 is a schematic structural diagram of an intelligent agent communication interaction apparatus according to an embodiment of this disclosure. As shown in Figure 10, the apparatus includes:

[0246] The first generation module 101 is configured to generate a trigger signal based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0247] The first determining module 102 is configured to determine the content of a multimodal merchant recommendation session using a trigger signal and pre-stored historical merchant recommendation dialogue data;

[0248] The first analysis module 103 is configured to perform conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0249] The first execution module 104 is configured to distribute the merchant recommendation action sequence to multiple intelligent agents for execution, and obtain the merchant recommendation execution result;

[0250] The first feedback module 105 is configured to send the results of the merchant recommendation to the target object.

[0251] Optionally, the first generation module 101 is further configured to: perform perception and understanding on the currently input multimodal information to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through the multimodal information; and generate a trigger signal based on the understanding result.

[0252] Optionally, the device further includes a sub-acquisition module configured to acquire historical dialogue data from a dialogue memory storage area, wherein the dialogue memory storage area is used to maintain multi-turn dialogue history information of the target object within a preset time range.

[0253] Optionally, the device further includes: a preprocessing module configured to preprocess historical dialogue data in response to receiving authorization from the target object to obtain a processing result, wherein the processing result is used to describe the target object's preference content; and storing the processing result in a long-term memory storage area, wherein the long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions, while the passive recording method extracts the target object's preference content from the target object's dialogue history information using an information extraction model.

[0254] Optionally, the first analysis module 103 is further configured to: identify the interaction intent of the target object from the multimodal session content through a semantic rule engine; in a known scenario, control the fine-tuned action planning model to generate a sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, control the zero-sample action planning model to generate a sequence of actions to be executed based on the interaction intent.

[0255] Optionally, the device further includes: a determination module configured to determine whether to use a multimodal search method to recall memory fragments of the interaction intent from the long-term memory storage area based on the interaction intent, wherein the long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions, while the passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragments are used to generate personalized responses in conjunction with the sequence of actions to be executed.

[0256] Optionally, the first execution module 104 is further configured to: assign the action sequence to be executed to modally matched agents among multiple agents for independent execution, and / or assign the action sequence to be executed to cross-modal agents among multiple agents for cooperative execution, thereby obtaining the execution result.

[0257] Optionally, the device further includes a storage module configured to store the execution results to a scene memory storage area, wherein the scene memory storage area is used to temporarily store the execution results of different actions of multiple intelligent agents.

[0258] Optionally, the first feedback module 105 is further configured to: determine whether all actions in the action sequence to be executed have been completed; and, in response to the completion of all actions, feed back the execution results to the target object using a streaming transmission method.

[0259] Optionally, the device further includes an expansion module configured to expand the number of agents and / or the multimodal processing capabilities of the multiple agents in response to the fulfillment of preset conditions.

[0260] By employing the embodiments of this disclosure, a trigger signal is generated based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object. Then, the multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data. Next, the multimodal conversation content is analyzed to obtain a sequence of actions to be executed. Then, the sequence of actions to be executed is assigned to multiple intelligent agents for execution to obtain the execution result. Finally, the execution result is fed back to the target object, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This achieves the technical effect of providing more natural, more efficient, and more personalized multimodal interaction for the needs of multiple scenarios, and solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple intelligent agents and cannot meet diverse scenario needs and personalized requirements.

[0261] It should be noted that the first generation module 101, the first determination module 102, the first analysis module 103, the first execution module 104, and the first feedback module 105 mentioned above correspond to steps S21 to S25 in the embodiments. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiments.

[0262] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described intelligent agent communication interaction method is also provided. Figure 11 is a schematic structural diagram of another intelligent agent communication interaction apparatus according to embodiments of this disclosure. As shown in Figure 11, the apparatus includes:

[0263] The second generation module 111 is configured to generate a trigger signal based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object;

[0264] The second determining module 112 is configured to determine the content of a multimodal merchant recommendation session using a trigger signal and pre-stored historical merchant recommendation dialogue data;

[0265] The second analysis module 113 is configured to perform conversation analysis on the multimodal merchant recommendation conversation content to obtain the merchant recommendation action sequence;

[0266] The second execution module 114 is configured to distribute the merchant recommendation action sequence to multiple intelligent agents for execution, and obtain the merchant recommendation execution result;

[0267] The second feedback module 115 is configured to send the merchant recommendation execution results back to the target object;

[0268] By employing embodiments of this disclosure, a trigger signal is generated based on the current input multimodal merchant recommendation request. The multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object. Then, the trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the multimodal merchant recommendation session content. Next, session analysis is performed on the multimodal merchant recommendation session content to obtain a merchant recommendation action sequence. This sequence is then distributed to multiple agents for execution, yielding merchant recommendation execution results. Finally, the execution results are fed back to the target object. This achieves efficient cross-modal multi-agent collaboration, providing a more natural, efficient, and personalized multimodal interaction for various scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0269] It should be noted that the second generation module 111, the second determination module 112, the second analysis module 113, the second execution module 114, and the second feedback module 115 mentioned above correspond to steps S41 to S45 in the embodiments. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors, and the above modules can also run in the server 10 provided in the embodiments.

[0270] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described intelligent agent communication interaction method is also provided. Figure 12 is a schematic structural diagram of another intelligent agent communication interaction apparatus according to embodiments of this disclosure. As shown in Figure 12, the apparatus includes:

[0271] The first acquisition module 121 is configured to acquire an agent communication interaction request through a first application programming interface. The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object.

[0272] The first return module 122 is configured to return an agent communication interaction response through a second application programming interface, wherein the response data carried in the agent communication interaction response includes: the execution result, which is obtained according to any of the above agent communication interaction methods.

[0273] By employing the embodiments of this disclosure, an agent communication interaction request is obtained through a first application programming interface (API). The request data carried in the agent communication interaction request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, an agent communication interaction response is returned through a second API. The response data carried in the agent communication interaction response includes an execution result, which is obtained according to any of the above agent communication interaction methods. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for the needs of multiple scenarios. It also solves the technical problem in related technologies that multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0274] It should be noted that the first acquisition module 121 and the first return module 122 mentioned above correspond to steps S51 to S52 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0275] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described intelligent agent communication interaction method is also provided. Figure 13 is a schematic structural diagram of another intelligent agent communication interaction apparatus according to an embodiment of this disclosure. As shown in Figure 13, the apparatus includes:

[0276] The second acquisition module 131 is configured to acquire the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, and the multimodal information is determined according to the multimodal interaction mode of the target object;

[0277] The second return module 132 is configured to respond to an agent communication interaction dialogue request and return an agent communication interaction dialogue response, wherein the information carried in the agent communication interaction dialogue response includes: the execution result, which is obtained according to any of the above agent communication interaction methods;

[0278] Display module 133 is configured to display the execution results within a graphical user interface.

[0279] By employing the embodiments of this disclosure, a current input agent communication interaction dialogue request is obtained. The request data carried in the agent communication interaction dialogue request includes multimodal information, which is determined based on the multimodal interaction method of the target object. Then, in response to the agent communication interaction dialogue request, an agent communication interaction dialogue reply is returned. The information carried in the agent communication interaction dialogue reply includes an execution result, which is obtained according to any of the aforementioned agent communication interaction methods. Finally, the execution result is displayed in a graphical user interface. This achieves the goal of efficient cross-modal multi-agent collaboration, thereby realizing the technical effect of providing more natural, efficient, and personalized multimodal interaction for needs in multiple scenarios. This solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0280] It should be noted that the second acquisition module 131, the second return module 132, and the display module 133 mentioned above correspond to steps S61 to S63 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0281] According to embodiments of this disclosure, another embodiment of an apparatus for implementing the above-described intelligent agent communication interaction method is also provided. Figure 14 is a schematic structural diagram of another intelligent agent communication interaction apparatus according to embodiments of this disclosure. As shown in Figure 14, the apparatus includes:

[0282] The first display module 141 is configured to display multimodal information on the operation interface in response to input commands applied to the operation interface.

[0283] The second display module 142 is configured to respond to processing instructions applied to the operation interface and display the execution result on the operation interface; wherein the execution result is obtained according to the intelligent agent communication interaction method of any one of claims 1 to 10.

[0284] By employing the embodiments of this disclosure, multimodal information is displayed on the operation interface in response to input commands applied to the operation interface, and then, in response to processing commands applied to the operation interface, the execution result is displayed on the operation interface. The execution result is obtained according to any of the aforementioned agent communication and interaction methods, thereby achieving the goal of efficient cross-modal multi-agent collaboration. This realizes the technical effect of providing more natural, efficient, and personalized multimodal interaction for needs in multiple scenarios, and solves the technical problem in related technologies where multimodal interaction systems lack efficient collaboration among multiple agents and cannot meet diverse scenario and personalized needs.

[0285] It should be noted that the first display module 141 and the second display module 142 mentioned above correspond to steps S71 to S72 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0286] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in the embodiments, but are not limited to the schemes provided in the embodiments.

[0287] Embodiments of this disclosure can provide a computing device. FIG15 is a structural block diagram of a computing device according to an embodiment of this disclosure. As shown in FIG15, the computing device A may include: one or more (only one is shown in FIG15) processors 1502, memory 1504, memory controller, and peripheral interfaces, wherein the peripheral interfaces may connect to radio frequency modules, audio modules, and displays, etc., without limitation.

[0288] The aforementioned computing device A can be understood as an integrated smart terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), all-in-one model machines, etc., and the computing device may have the model described in the above embodiments of this disclosure pre-installed.

[0289] Specifically, computing device A can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, computing device A can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, computing device A also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, computing device A can also create applications based on models, providing API calling capabilities, allowing models to be called into created applications through API interfaces, and providing application management tools for application management and monitoring.

[0290] Furthermore, the computing device A can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0291] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0292] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0293] It will be understood by those skilled in the art that the structure shown in FIG15 is merely illustrative, and computing device A may also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. FIG15 does not limit the structure of the aforementioned computing device. For example, computing device A may include more or fewer components (such as network interfaces, display devices, etc.) than shown in FIG15, or may have a different configuration than that shown in FIG15.

[0294] Embodiments of this disclosure can provide an electronic device. FIG16 is a structural block diagram of an electronic device according to an embodiment of this disclosure. As shown in FIG16, the electronic device may include: an input / output device 162; a memory 164; and a processor 166, wherein the processor 166 is connected to the input / output device 162 and the memory 164 via a bus 168.

[0295] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0296] The processor can invoke an executable program stored in memory via a transmission device to execute the method described in any of the above embodiments.

[0297] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0298] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the intelligent agent communication and interaction method provided in the above embodiments.

[0299] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0300] Embodiments of this disclosure also provide a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described intelligent agent communication and interaction methods.

[0301] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0302] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0303] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0304] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0305] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0306] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A method for intelligent agent communication and interaction, comprising: A trigger signal is generated based on the current input multimodal information, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; The multimodal conversation content is determined using the trigger signal and pre-stored historical dialogue data; The multimodal conversation content is analyzed to obtain a sequence of actions to be executed; The sequence of actions to be executed is assigned to multiple agents for execution, and the execution results are obtained. The execution result is then fed back to the target object.

2. The intelligent agent communication and interaction method according to claim 1, wherein, Generating the trigger signal based on the currently input multimodal information includes: The multimodal information currently input is perceptually understood to obtain an understanding result, wherein the understanding result is used to perceive the intention or action of the target object through the multimodal information; The trigger signal is generated based on the understanding results.

3. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The intelligent agent communication and interaction method further includes: The historical dialogue data is obtained from the dialogue memory storage area, wherein the dialogue memory storage area is used to maintain the historical dialogue information of the target object within a preset time range.

4. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The intelligent agent communication and interaction method further includes: In response to receiving authorization from the target object, the historical dialogue data is preprocessed to obtain a processing result, wherein the processing result is used to describe the target object's preference content; The processing result is stored in the long-term memory storage area.

5. The intelligent agent communication and interaction method according to claim 4, wherein, The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object, through active recording or passive recording. The active recording method records the target object's preference content according to the target object's instructions, while the passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information.

6. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The multimodal session content is analyzed to obtain the sequence of actions to be executed, which includes: The semantic rule engine identifies the interaction intent of the target object from the multimodal session content; In a known scenario, the finely tuned action planning model generates the sequence of actions to be executed based on the interaction intent; or, in an unknown scenario, the zero-sample action planning model generates the sequence of actions to be executed based on the interaction intent.

7. The intelligent agent communication and interaction method according to claim 6, wherein, The intelligent agent communication and interaction method further includes: Based on the interaction intent, it is determined whether to use a multimodal search method to recall the memory fragment of the interaction intent from the long-term memory storage area. The long-term memory storage area is used to maintain the target object's preference content from the perspective of the target object through an active recording method or a passive recording method. The active recording method records the target object's preference content according to the target object's instructions. The passive recording method uses an information extraction model to extract the target object's preference content from the target object's dialogue history information. The memory fragment is used to generate a personalized response in conjunction with the action sequence to be executed.

8. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The sequence of actions to be executed is assigned to the multiple agents for execution, and the execution results include: The sequence of actions to be executed is assigned to modality-matching agents among the multiple agents for independent execution, and / or the sequence of actions to be executed is assigned to cross-modality agents among the multiple agents for cooperative execution, thereby obtaining the execution result.

9. The intelligent agent communication and interaction method according to claim 8, wherein, The intelligent agent communication and interaction method further includes: The execution results are stored in the scene memory storage area, which is used to temporarily store the execution results of different actions of the multiple intelligent agents.

10. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, Feeding the execution result back to the target object includes: Determine whether all actions in the sequence of actions to be executed have been completed; In response to the completion of all the aforementioned actions, the execution results are fed back to the target object using a streaming transmission method.

11. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The intelligent agent communication and interaction method further includes: In response to the fulfillment of preset conditions, the number of intelligent agents and / or the multimodal processing capabilities of the multiple intelligent agents are expanded.

12. The intelligent agent communication and interaction method according to claim 1 or 2, wherein, The multimodal session content includes at least one of the following: multimodal information, current session context determined based on historical dialogue data, and user device information.

13. A method for intelligent agent communication and interaction, comprising: A trigger signal is generated based on the current input multimodal merchant recommendation request, wherein the multimodal merchant recommendation request is determined according to the multimodal interaction method of the target object; The trigger signal and pre-stored historical merchant recommendation dialogue data are used to determine the content of the multimodal merchant recommendation session; The multimodal merchant recommendation conversation content is analyzed to obtain the merchant recommendation action sequence; The merchant recommendation action sequence is distributed to multiple intelligent agents for execution, and the merchant recommendation execution result is obtained. The results of the merchant recommendation will be fed back to the target object.

14. A method for intelligent agent communication and interaction, comprising: The intelligent agent communication interaction request is obtained through the first application programming interface, wherein the request data carried in the intelligent agent communication interaction request includes: multimodal information, which is determined according to the multimodal interaction mode of the target object; The agent communication interaction response is returned through the second application programming interface, wherein the response data carried in the agent communication interaction response includes: the execution result, which is obtained according to the agent communication interaction method according to any one of claims 1 to 11.

15. A method for intelligent agent communication and interaction, comprising: Obtain the currently input intelligent agent communication interaction dialogue request, wherein the request data carried in the intelligent agent communication interaction dialogue request includes: multimodal information, wherein the multimodal information is determined based on the multimodal interaction mode of the target object; In response to the agent communication interaction dialogue request, an agent communication interaction dialogue response is returned, wherein the information carried in the agent communication interaction dialogue response includes: an execution result, the execution result being obtained according to the agent communication interaction method according to any one of claims 1 to 12; The execution results are displayed within the graphical user interface.

16. A method for intelligent agent communication and interaction, comprising: In response to input commands applied to the user interface, multimodal information is displayed on the user interface. In response to processing instructions applied to the operation interface, the execution result is displayed on the operation interface; The execution result is obtained according to the intelligent agent communication and interaction method described in any one of claims 1 to 12.

17. An intelligent agent communication and interaction system, comprising: The perception layer is used to generate trigger signals based on the multimodal information of the current input, wherein the multimodal information is determined according to the multimodal interaction mode of the target object; The control layer is used to determine the multimodal session content using the trigger signal and pre-stored historical dialogue data; The planning layer is used to perform session analysis on the multimodal session content to obtain a sequence of actions to be executed; The execution layer is used to distribute the sequence of actions to be executed to multiple agents for execution, and obtain the execution result. The planning layer is also used to feed back the execution results to the target object.

18. An intelligent agent communication and interaction system, comprising: A client is used to send multimodal information, wherein the multimodal information is determined based on the multimodal interaction method of the target object; The server, connected to the client, is used to generate a trigger signal based on the multimodal information, determine the multimodal conversation content using the trigger signal and pre-stored historical dialogue data, perform conversation analysis on the multimodal conversation content to obtain a sequence of actions to be executed, and assign the sequence of actions to be executed to multiple agents for execution to obtain execution results. The client is also used to output the execution result.

19. An electronic device comprising: Memory, which stores executable programs; A processor for running the program, wherein the program executes the intelligent agent communication and interaction method according to any one of claims 1 to 16 when it runs.

20. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the intelligent agent communication interaction method according to any one of claims 1 to 16.