Navigation method and device

Abstract

A navigation method and device which are applied to the field of artificial intelligence and are used for introducing global information, fusing the global information with a current observation image, and then performing reasoning on the fused information by means of a multimodal large language model to obtain an action decision, so as to achieve high-quality navigation of an agent. The method comprises: acquiring a first observation image of an agent (step 301); on the basis of a task inputted by a user, determining cue information from global information (step 302), wherein the global information is used for indicating memory information stored in the agent, the memory information is used for describing an environment, the memory information is obtained on the basis of a second observation image, the second observation image is captured by the agent visiting the environment, and the cue information is used for assisting the agent in completing the task; fusing the cue information with the first observation image to obtain fused information (step 303); and performing reasoning on the fused information by means of a multimodal large language model to obtain an action decision (step 304), wherein the action decision is used for instructing the agent to execute the task.

Description

A navigation method and device

[0001] This application claims priority to Chinese Patent Application No. 202510186871.5, filed on February 19, 2025, entitled "A Navigation Method and Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a navigation method and device. Background Technology

[0003] Embodied navigation is a crucial component of embodied artificial intelligence (EAI), with wide applications in scenarios such as home and office services, logistics and delivery, and factory inspection. EAI has two key characteristics: First, it faces highly diverse scenarios. Compared to autonomous driving, EAI needs to handle more complex spatial layouts, such as indoor spaces, factory areas, and shopping malls, where the uncertainty is significant. Second, it demands a high degree of autonomy. Embodied intelligent agents sometimes need to efficiently explore and execute commands in unfamiliar environments, placing high demands on the autonomy of navigation algorithms. Furthermore, due to the rapid development of large language models (LLMs) in recent years, their application in embodied navigation has become a mainstream trend in both industry and academia.

[0004] Currently, existing solutions score actions based on the first-person perspective images (or current observation perspective) of a robot or other intelligent agent, and update the scoring results to a global map. This global map records the observed image regions and their scores. Action decisions are then made based on this global map, and the robot or other intelligent agent executes these decisions to complete the task. Alternatively, a global semantic map can be constructed based on the first-person perspective images. Candidate point descriptions relevant to the task can be identified from the global semantic map, and action decisions can be derived using a large language model.

[0005] However, existing solutions only score the agent's first-person perspective image and make action decisions based on the scores, without considering global information, making them prone to getting stuck in local optima or suboptimal solutions. On the other hand, approaches based on global semantic maps rely on language descriptions for abstract description and reasoning, losing high-dimensional semantic information about the environment and limiting the model's capabilities in complex spatial environments and task scenarios. Therefore, how to construct efficient embodied navigation decision-making remains a pressing problem to be solved. Summary of the Invention

[0006] This application provides a navigation method and device for introducing global information, fusing the global information with the currently observed image, and then using a multimodal large language model to reason about the fused information to obtain action decisions, so as to achieve high-quality navigation for intelligent agents.

[0007] In a first aspect, this application provides a navigation method, comprising: firstly acquiring a first observation image of an intelligent agent, wherein the first observation image is a partial image of a scene captured by the intelligent agent through a camera; subsequently, determining cue information from global information based on a task input by a user, wherein the global information is used to indicate memory information stored by the intelligent agent, the memory information is used to describe the scene, the memory information is obtained based on a second observation image, wherein the second observation image is captured by the intelligent agent while visiting the scene, and the cue information is used to assist the intelligent agent in completing the task; fusing the cue information with the first observation image to obtain fused information; and then reasoning on the fused information through a multimodal large language model to obtain an action decision, the action decision being used to instruct the intelligent agent to perform the task.

[0008] In this embodiment, additional global information is introduced, including memory information obtained by the agent from observing visited areas, where the memory information describes the scene. Subsequently, the agent can determine task-related clues from the global information, enabling it to make action decisions based on this information. This avoids the problem of getting trapped in local optima or suboptimal solutions when making action decisions solely based on the current observed image (the first observed image). Furthermore, the clue information and the first observed image are fused, integrating global and local information, and combining textual and visual information to obtain richer task-related information, thereby achieving higher-quality action decisions.

[0009] In one possible implementation, the aforementioned process of fusing the clue information with the first observation image to obtain fused information may include: labeling the clue information onto the first observation image to obtain fused information.

[0010] In this embodiment, global and local information are fused to avoid the problem of local optimal or suboptimal solutions caused by making action decisions based solely on the current viewpoint image.

[0011] In one possible implementation, the aforementioned step of annotating the clue information onto the first observation image to obtain fused information may include: calculating the position information of the clue in the first observation image based on the clue information, the position information being used to indicate the position of the clue information in the first observation image; and annotating the clue information onto the first observation image based on the position information to obtain fused information.

[0012] In this embodiment, the clue information can be labeled onto the first observation image based on its position information in the first observation image, so as to achieve the fusion of global and local information.

[0013] In one possible implementation, the aforementioned step of annotating the cue information onto the first observation image to obtain fused information may include: calculating a path based on the cue information, the path being used to indicate the direction of the cue information relative to the agent; projecting the path onto the first observation image to obtain a fused image; and annotating the fused image based on the cue information to obtain fused information.

[0014] In this embodiment, a path can be obtained based on clue information, and the path can be mapped to the fusion information obtained from the first observation image, which can provide more intuitive information and thus improve the accuracy of action decision-making based on fusion information.

[0015] In one possible implementation, before determining the clue information from global information based on the task input by the user, the method may further include: determining whether the task has been completed based on the first observed image, and obtaining a determination result.

[0016] In one possible implementation, the aforementioned determination of clue information from global information based on the user-input task may include: when the judgment result indicates that the task is not completed, determining clue information from global information based on the task.

[0017] In one possible implementation, the aforementioned determination of clue information from global information based on the task may include: using information with a correlation higher than a preset value as clue information based on the correlation between the task and global information; and / or, obtaining clue information by reasoning about the task through preset rules or a large language model.

[0018] In one possible implementation, the method may further include: extracting memory information from the first observed image to obtain local information; and updating global information based on the local information.

[0019] Secondly, this application proposes a navigation device, comprising: an acquisition module for acquiring a first observation image of an intelligent agent, wherein the first observation image is a partial image of a scene captured by the intelligent agent through a shooting device; a determination module for determining clue information from global information based on a task input by a user, wherein the global information is used to indicate memory information stored by the intelligent agent, the memory information is used to describe the scene, the memory information is obtained based on a second observation image, wherein the second observation image is captured by the intelligent agent while visiting the scene, and the clue information is used to assist the intelligent agent in completing the task; a fusion module for fusing the clue information with the first observation image to obtain fused information; and a reasoning module for reasoning on the fused information through a multimodal large language model to obtain an action decision, wherein the action decision is used to instruct the intelligent agent to perform the task.

[0020] In one possible implementation, the aforementioned fusion module is specifically used to: annotate the clue information onto the first observation image to obtain fused information.

[0021] In one possible implementation, the aforementioned fusion module is specifically used to: calculate the position information of the clue information on the first observation image based on the clue information, the position information being used to indicate the position of the clue information in the first observation image; and annotate the clue information onto the first observation image based on the position information to obtain fused information.

[0022] In one possible implementation, the aforementioned fusion module is specifically used to: calculate a path based on the cue information, the path being used to indicate the direction of the cue information relative to the agent; project the path onto a first observation image to obtain a fused image; and annotate the fused image based on the cue information to obtain fused information.

[0023] In one possible implementation, before determining the clue information from the global information based on the user-input task, the device may further include: a judgment module, used to judge whether the task is completed based on the first observation image, and obtain a judgment result.

[0024] In one possible implementation, the aforementioned determining module is specifically used to: when the judgment result indicates that the task is not completed, determine the clue information from the global information based on the task.

[0025] In one possible implementation, the aforementioned determining module is specifically used to: take information with a correlation higher than a preset value as clue information based on the correlation between the task and global information; and / or, obtain clue information by reasoning about the task through preset rules or a large language model.

[0026] In one possible implementation, the device may further include: an extraction module for extracting memory information from the first observed image to obtain local information; and an update module for updating global information based on the local information.

[0027] Thirdly, this application provides a navigation device, which includes: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement the method as described in the first aspect or any implementation thereof.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0029] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0030] The technical effects of the second to fifth aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0031] Figure 1 is a schematic diagram of an artificial intelligence main framework applied in this application;

[0032] Figure 2 is a schematic diagram of a system architecture provided in this application;

[0033] Figure 3 is a flowchart illustrating a navigation method provided in this application;

[0034] Figure 4 is a flowchart illustrating the process of determining clue information from global information based on the task.

[0035] Figure 5 shows an example of fusing clue information and observation images;

[0036] Figure 6 is an example diagram of navigation exploration based on the task of finding a water cup;

[0037] Figure 7 is an example diagram of navigation exploration based on coffee machine recognition tasks;

[0038] Figure 8 is a schematic diagram of the structure of a navigation device provided in this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0040] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0041] First, the overall workflow of an artificial intelligence system is described, as shown in Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The framework is then elaborated on from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0042] (1) Infrastructure

[0043] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0044] (2) Data

[0045] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0046] (3) Data processing

[0047] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0048] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0049] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0050] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0051] (4) General ability

[0052] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0053] (5) Smart Products and Industry Applications

[0054] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0055] To facilitate understanding, some concepts involved in this application will be introduced first.

[0056] 1. Embodied Artificial Intelligence (EAI)

[0057] Embodied intelligence is an intelligent system that perceives and acts based on physical entities. It acquires information, understands problems, makes decisions, and executes actions through the interaction between the intelligent agent and its environment, thereby exhibiting intelligent behavior and adaptability.

[0058] 2. Embodied Navigation

[0059] Embodied navigation is a technology that enables intelligent agents to complete navigation tasks in a 3D environment through senses such as vision and hearing, combined with language, movement, and interactive behaviors. It involves the entire process from perceiving the environment and understanding instructions to executing actions, aiming to simulate human navigation capabilities in the real world.

[0060] 3. Large Language Model (LLM)

[0061] LLM (Language Modeling) refers to a language model with hundreds of millions (or more) parameters trained on massive amounts of text data. It's a deep learning-based natural language processing model. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and scalability. For example, in movie recommendation scenarios, large language models can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar films. Large language models can also generate recommendation reasons; for instance, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the products.

[0062] 4. Multimodal Large Language Model (MLLM)

[0063] A multimodal large language model is an artificial intelligence model capable of processing and understanding multiple types of data (such as text, images, audio, video, etc.). It combines techniques such as natural language processing (NLP) and computer vision (CV) to extract information from data of different modalities and generate cross-modal output.

[0064] 5. Visual Language Model (VLM)

[0065] Visual language models are broadly defined as multimodal models capable of learning from images and text, taking images and text as input and generating text or images as output. Large visual language models perform well in zero-shot learning, have good generalization ability, and can handle various types of images, including documents and web pages.

[0066] 6. Intelligent Agent

[0067] An intelligent agent is an entity that can perceive, make decisions, and execute actions in its environment to achieve specific goals. It possesses autonomy, responsiveness, goal orientation, and learning capabilities, and can operate independently in complex environments. Examples include self-driving cars, smart home systems, and robots.

[0068] 7. Frontier Map Technology

[0069] Frontier mapping is a common navigation and environmental exploration method used in robots and autonomous vehicles. It is primarily used to build maps in unknown or partially known environments and identify unexplored areas or "frontiers" to be explored next. Frontier mapping can quickly identify areas that should be prioritized for exploration; the key is selecting appropriate boundaries as the next navigation target to improve navigation and exploration efficiency.

[0070] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0071] Referring to Figure 2, a system architecture 200 provided in this application is shown. As shown in Figure 2, the data acquisition device 260 can be used to collect training data. After the data acquisition device 260 collects the training data, it stores the training data in the database 230. The training device 220 trains a multimodal large language model 202 based on the training data maintained in the database 230.

[0072] It should be noted that in practical applications, the training data maintained in database 230 may not all come from the data acquisition device 260; it may also be received from other devices. Furthermore, it should be noted that training device 220 may not necessarily train the multimodal large language model 202 entirely based on the training data maintained in database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0073] The computing module 211 includes a fusion module 201 and a multimodal large language model 202. The navigation method mentioned in this application can be executed by the computing module 211. The fusion module 201 is used to obtain fused information based on the user-input task and the first observation image of the acquired agent. The multimodal large language model 202 is used to reason about the fused information to obtain action decisions. The navigation method proposed in this application can be applied to different devices, such as the execution device 210 shown in Figure 2. The execution device 210 can be an intelligent vehicle, a mobile robot, or other intelligent agent. In Figure 2, the execution device 210 is equipped with a transceiver 212, which may include an input / output (I / O) interface or other wireless or wired communication interfaces for data interaction with external devices. Taking the I / O interface as an example, the user can input data to the I / O interface through the client device 240. For example, in the following embodiments of this application, the client device 240 can send the required task to the execution device 210.

[0074] During the preprocessing of input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0075] It is worth noting that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of this application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0076] Currently, with the rapid development of large language models, applying them to embodied navigation has become a research trend. However, existing solutions mainly rely on first-person perspective images of the agent for action decisions, or on constructing a global semantic map based on first-person perspective images for action decisions. However, making action decisions solely based on the current perspective is prone to getting stuck in local optima or suboptimal solutions, while constructing a global semantic map based on first-person perspective images and subsequently relying on this global semantic map for abstract description and reasoning through language loses the ability to understand complex spaces.

[0077] Based on this, this application proposes a navigation method that introduces global information. According to the user-input task, clue information (task-related information) is determined from the global information. This clue information is then fused with the agent's first observation image. A multimodal large language model is used to reason about the fused information to obtain an action decision, which can then be used to complete the user-input task. The introduction of global information avoids making action decisions solely based on the agent's current perspective, thus preventing getting trapped in local optima or engaging in repetitive exploration. Furthermore, fusing clue information with the observation image provides multimodal information, improving the quality of action decisions.

[0078] The navigation method provided in this application embodiment can be widely applied to intelligent agents such as mobile robots, autonomous vehicles, and drones, and can also be widely applied to scenarios such as home assistants, warehouse management, retail guidance, hospital navigation, elderly care, smart offices, visitor guidance, and emergency response.

[0079] For example, embodied intelligence-based navigation robots can plan routes indoors and help complete tasks such as housework. Robots with accurate navigation provide a more efficient, reliable, and interactive user experience; they can automatically navigate, inventory, and manage storage in warehouse environments, reducing human error and improving efficiency; they can guide customers to the products they need in stores, providing recommendations and promotional information; they can help patients or visitors find wards, doctors' offices, or other facilities in hospitals, improving patient satisfaction; nursing robots can autonomously navigate facilities, providing the support and assistance needed by the elderly, improving the quality of care; robots can deliver documents, beverages, or other items in office buildings, simplifying internal logistics; they can also provide navigation and information retrieval services for visitors, improving the visitor experience; furthermore, in emergencies such as earthquakes and fires, embodied intelligence navigation enables robots to quickly navigate to dangerous areas for search and rescue, fire detection, and firefighting tasks, improving emergency response efficiency and safety.

[0080] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0081] Referring to Figure 3, a flowchart of a navigation method provided in this application is shown below.

[0082] Step 301: Obtain the first observation image of the agent;

[0083] The first observation image is a partial image of the scene captured by the agent using a camera based on its current pose. For example, if the agent is a mobile robot, the first observation image is the current viewpoint image obtained by the mobile robot through its camera. The first observation image provides visual information for the agent to complete the user-inputted task.

[0084] Step 302: Based on the task input by the user, determine the clue information from the global information;

[0085] In this embodiment of the application, the task input by the user can be obtained first. For example, the task input by the user can be "Please help me find a water cup", or "Please find a coffee machine", etc., and the specific task is not limited here.

[0086] Optionally, after the agent acquires the first observation image, it can determine whether the user-input task has been completed based on the first observation image and obtain the judgment result.

[0087] Specifically, the first observed image can be analyzed using an object detection model or a multimodal large language model to obtain the judgment result.

[0088] In this embodiment of the application, in order to avoid the problem of getting stuck in local optima or suboptimal solutions due to predicting action decisions based solely on the agent's current observed image, global information can be introduced. This global information is used to indicate the memory information stored by the agent, which is used to describe the scene.

[0089] Specifically, when an agent visits a scene, it can move continuously and take pictures of the scene, thereby obtaining multiple different second observation images. Each second observation image corresponds to a visited area, and the agent can extract, convert, and store the memory information in the visited area (or second observation image) to obtain global information, which includes the memory information of the visited area.

[0090] Optionally, memory information can be extracted from the first observed image to obtain local information; subsequently, the global information is updated based on the local information. In this embodiment, the global information can be incrementally updated continuously, updating the global information with new memory information from the observed images acquired by the agent, so that the global information includes more scene-related information. The new memory information is information different from the memory information stored in the global information.

[0091] The global information can take various representations, such as 2D / 3D semantic mesh maps, semantic graphs, neural radiance fields (NeRF), and 3D Gaussian splashing (3DGS), without specific limitations. It is understood that the memory information extracted from the visited region can be in the form of 2D / 3D semantic mesh maps, semantic graphs, NeRF, 3DGS, etc., and this memory information includes the scene semantics of the visited region and / or the location information of the visited region. The scene semantics includes the category and location information of objects in the visited region, and a description of the spatial environment surrounding the objects.

[0092] Optionally, when the judgment result indicates that the user-input task has not been completed, clue information can be obtained from the global information based on the task.

[0093] Because the global information may be stored in a 2D map or other abstract data format, it cannot be directly fused with the observed image. Furthermore, the global information summary may contain redundant information irrelevant to the task. Therefore, before determining clues from the global information, it can be parsed to obtain parsed global information. This parsed global information includes scene semantics of the visited areas and their location information. Scene semantics describes the visited areas.

[0094] Optionally, clues can be determined from the parsed global information based on the task. Alternatively, clues can be determined from the global information first, and then parsed. The specific approach is not limited here.

[0095] In this embodiment, the clue information is used to assist the intelligent agent in completing the user-inputted task. Specifically, the clue information is information related to the user-inputted task within the global information. For example, the clue information may include the category and location information of the task-related object, candidate point information, visited areas, or a description of the spatial environment of the scene. The candidate point information includes the name or location of the candidate object. For instance, if the user inputs a task to find a water glass, based on this task, clue information related to finding the water glass can be determined from the global information. This clue information might include the kitchen and its location. The intelligent agent can then move to the kitchen to observe it until it finds the water glass.

[0096] Optionally, information in the global information with a correlation higher than a preset value can be used as clue information based on the correlation between the task and the global information; and / or, clue information can also be obtained by reasoning about the task through preset rules or a large language model.

[0097] In this process, clues are obtained by reasoning about the task using pre-defined rules or a large language model. These clues are typically potential candidate points. For example, when the task is to find a coffee machine, based on the task, the pre-defined rules or a large language model can infer that the candidate point is the kitchen. The pre-defined rules include mapping relationships between multiple objects and locations, so corresponding candidate points can be inferred from the objects mentioned in the task.

[0098] For example, Figure 4 is a flowchart illustrating the process of determining clue information from global information based on a task. As shown in Figure 4, scene semantics and access memory can be parsed from the global information first, where access memory includes location information of visited areas. Subsequently, task-related clue information can be determined based on scene semantics. This clue information includes the category and coordinates of associated objects, candidate point information, and visited area information. For example, associated objects could be a monitor (102,1), a desk (23,142), and a laptop (102,10). Furthermore, based on scene semantics and the task, candidate points and their weight values, location information, or surrounding descriptions of candidate points are obtained through large language models or pre-defined rules. The location information of visited areas can be obtained based on access memory.

[0099] Step 303: Fuse the clue information and the first observation image to obtain fused information;

[0100] In this embodiment, the clue information is determined from the global information, and the first observation image is a local image of the scene obtained by the agent when accessing the scene. Therefore, by fusing the clue information and the first observation image, the global information and the agent's current observation image can be fused, thereby solving the problem of getting stuck in a local optimum or suboptimal solution due to determining the action decision only based on the current observation image.

[0101] There are several ways to fuse the clue information and the agent's first observation image, and we will not limit the specific methods here. The following sections will introduce the possible methods for fusing the clue information and the first observation image.

[0102] Optionally, the clue information can be directly annotated onto the first observation image to obtain fused information. The clue information can be annotated onto the first observation image in the form of text or images.

[0103] Optionally, the position information of the clue information on the first observation image can be calculated based on the clue information. This position information is used to indicate the position of the clue information in the first observation image. Subsequently, the clue information can be labeled onto the first observation image based on the position information to obtain fused information.

[0104] For example, the clue information includes the category and location of the task-related object and the candidate points and their location information. Based on this clue information, the task-related object and the candidate points can be marked at their corresponding positions in the first observation image.

[0105] Taking candidate points as an example, the candidate point's index and weight score can be labeled to the corresponding position in the first observation image based on the calculated location information. When a candidate point exists in the first observation image (current observation field), the location information indicates the candidate point's position in the first observation image; when a candidate point does not exist in the first observation image, the location information indicates the candidate point's position direction.

[0106] Optionally, a path can be calculated based on the cue information, which indicates the direction of the cue information relative to the agent; then the path can be projected onto the first observation image to obtain a fused image; after obtaining the fused image, the fused image can be labeled to obtain fused information.

[0107] Since the task-related objects or candidate points in the clue information may or may not exist in the first observation image, when the task-related object does not exist in the first observation image, the generated path can provide the agent with a direction for the task-related object based on the location information of the task-related object and the first observation image, so that the agent can find the task-related object according to the path.

[0108] For example, Figure 5 is an example of fusing clue information and observation image. As can be seen from Figure 5, there is no coffee table in the observation image. Based on the observation image and the location information of the coffee table, a path can be generated to indicate the direction of the coffee table. Furthermore, text information (such as "coffee table") can be labeled next to the path to indicate that the task-related object pointed to by the path is the coffee table.

[0109] Specifically, since the clue information includes task-related objects or candidate points and their location information, the path from the agent to the task-related object or candidate point can be calculated by combining the spatial environment description in the clue information. Then, this path is projected onto the first observation image to obtain a fused image. The fused image can then be labeled using text or images, etc., without specific limitations here.

[0110] For example, if the task is to find a water glass, the first clues determined from the global information include the task-related objects being the kitchen and the table. Based on the location information of the kitchen and the table, as well as the relevant spatial environment description, such as the table being next to the sofa in the living room, the path for the agent to move to the kitchen or the table based on its current location can be calculated. The path can be labeled with text or images next to it, indicating that the path leads to the kitchen or the table.

[0111] Step 304: Reason about the fused information using a multimodal large language model to obtain action decisions.

[0112] The multimodal large language model can be a vision language model (VLM) or other multimodal large language models; no specific limitation is made here.

[0113] After receiving the action decision, the agent can execute the action according to the action decision until the user inputs the task.

[0114] In this embodiment, the fused information includes text data and image data. By reasoning about the multimodal fused information through a multimodal large language model, the textual and visual information are fully utilized, thereby obtaining richer scene information and improving the quality of the action decisions obtained through reasoning.

[0115] For example, based on the HSSD scene small object search dataset, the navigation method provided in this application and the current state-of-the-art (SOTA) model InstructNav are used to navigate to the user-input task. The navigation method provided in this application outperforms the SOTA in terms of navigation success rate, success-weighted path length (SPL) metric, and average number of steps to complete the task, thus indicating that the navigation method proposed in this application is more efficient in item search. The SPL metric reflects the optimality of the path.

[0116] For example, taking navigation and exploration of a target object in an unknown scene as an example, the robot acquires a first-person panoramic view image (i.e., the agent's current observation image) and simultaneously constructs a frontier map by converting RGBD image data into point cloud data. Then, points that the robot can move to and are close to the frontiers are selected as candidate markers. Next, a visual language model describes the surrounding environment of each marker, and information about different markers is stored in global memory. Furthermore, historical markers can be mapped to the first-person view image to avoid repeated exploration. Afterward, the VLM (Visual Mapping Model) can mark the optimal marker point from the first-person panoramic image, and the robot uses its local navigation capabilities to move to that marker point until the target object is found. Moreover, if the VLM determines that there are no suitable marker points to explore in the current first-person panoramic image, it can determine unexplored candidate marker points relevant to the task from global memory.

[0117] The VLM model can also be used for other existing multimodal large language models, or for open-source large models that undergo task-specific supervised fine-tuning (SFT). Furthermore, when the ID of a candidate point is incorrectly determined using a multimodal large language model, to ensure the navigation task can proceed, boundaries or candidate points can be randomly determined from global information as a backup exploration method.

[0118] As shown in Figures 6 and 7, Figure 6 is an example of navigation exploration based on the task of finding a water cup, and Figure 7 is an example of navigation exploration based on the task of recognizing a coffee machine. As shown in Figure 6, based on the user's input task "find a water cup", and based on the VLM model or preset rules, it is inferred that the water cup may be in the kitchen, bedroom, or dining area. Combining the candidate points marked in the robot's first-person global view image, mark point 10 can be determined as the optimal mark point. Based on this mark point, the kitchen can be found, and finally, the water cup can be found in the kitchen.

[0119] As shown in Figure 7, the user inputs the task of "finding the coffee machine". Based on VLM reasoning or preset rules, it is determined that the coffee machine may be in the kitchen. Combined with the robot's current observation image, the image indicates that candidate point 8 (refrigerator) may be in the kitchen. Therefore, candidate point 8 can be determined as the optimal candidate point from multiple candidate points, and the action decision is obtained to instruct the robot to move to candidate point 8. Finally, after the robot moves to candidate point 8, it can find the coffee machine.

[0120] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0121] Referring to Figure 8, a schematic diagram of the structure of a navigation device provided in this application is as follows.

[0122] The acquisition module 801 is used to acquire the first observation image of the intelligent agent, which is a partial image of the scene captured by the intelligent agent through the imaging device;

[0123] The determination module 802 is used to determine the clue information from the global information based on the task input by the user. The global information is used to indicate the memory information stored by the agent. The memory information is used to describe the scene. The memory information is obtained based on the second observation image. The second observation image is obtained by the agent when it visits the scene and takes pictures. The clue information is used to assist the agent in completing the task.

[0124] The fusion module 803 is used to fuse the clue information with the first observation image to obtain fused information;

[0125] The reasoning module 804 is used to reason about the fused information through a multimodal large language model to obtain action decisions, which are used to instruct the agent to perform tasks.

[0126] In one possible implementation, the aforementioned fusion module 803 is specifically used to: annotate the clue information onto the first observation image to obtain fused information.

[0127] In one possible implementation, the aforementioned fusion module 803 is specifically used to: calculate the position information of the clue information on the first observation image based on the clue information, the position information being used to indicate the position of the clue information in the first observation image; and annotate the clue information onto the first observation image based on the position information to obtain fused information.

[0128] In one possible implementation, the aforementioned fusion module 803 is specifically used to: calculate a path based on the cue information, the path being used to indicate the direction of the cue information relative to the agent; project the path onto the first observation image to obtain a fused image; and annotate the fused image based on the cue information to obtain fused information.

[0129] In one possible implementation, the aforementioned fusion module 803 is specifically used to: annotate the clue information onto the first observation image to obtain fused information.

[0130] In one possible implementation, before determining the clue information from the global information based on the user-input task, the device may further include: a judgment module 805, used to judge whether the task is completed based on the first observation image and obtain a judgment result.

[0131] In one possible implementation, the aforementioned determining module 802 is specifically used to: when the determination result indicates that the task is not completed, determine the clue information from the global information according to the task.

[0132] In one possible implementation, the aforementioned determining module 802 is specifically used to: based on the correlation between the task and global information, identify information with a correlation higher than a preset value as clue information; and / or, obtain clue information by reasoning about the task through preset rules or a large language model.

[0133] In one possible implementation, the device may further include: an extraction module 806 for extracting memory information from the first observation image to obtain local information; and an update module 807 for updating global information based on the local information.

[0134] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps of the method described in the embodiment shown in FIG3 above.

[0135] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method steps described in the embodiment shown in FIG3 above.

[0136] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0143] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A navigation method, characterized in that, include: Acquire a first observation image of the intelligent agent, wherein the first observation image is a partial image of the scene captured by the intelligent agent through a shooting device; Based on the task input by the user, clue information is determined from global information. The global information is used to indicate the memory information stored by the agent. The memory information is used to describe the scene. The memory information is obtained based on a second observation image, which is taken by the agent when it visits the scene. The clue information is used to assist the agent in completing the task. The clue information is fused with the first observed image to obtain fused information; The fused information is reasoned through a multimodal large language model to obtain action decisions, which are used to instruct the agent to perform the task.

2. The method according to claim 1, characterized in that, The step of fusing the clue information with the first observed image to obtain fused information includes: The clue information is labeled onto the first observation image to obtain the fused information.

3. The method according to claim 2, characterized in that, The step of annotating the clue information onto the first observed image to obtain the fused information includes: Based on the clue information, the position information of the clue information on the first observation image is calculated, and the position information is used to indicate the position of the clue information in the first observation image; Based on the location information, the clue information is labeled onto the first observation image to obtain the fused information.

4. The method according to claim 2, characterized in that, The step of annotating the clue information onto the first observed image to obtain the fused information includes: Based on the clue information, a path is calculated, and the path is used to indicate the direction of the clue information relative to the agent; The path is projected onto the first observed image to obtain a fused image; Based on the clue information, the fused image is annotated to obtain the fused information.

5. The method according to any one of claims 1 to 4, characterized in that, Before determining the clue information from global information based on the user-input task, the method further includes: Based on the first observed image, determine whether the task has been completed, and obtain the determination result.

6. The method according to claim 5, characterized in that, The step of determining clue information from global information based on the user-input task includes: When the judgment result indicates that the task is not completed, the clue information is determined from the global information based on the task.

7. The method according to claim 6, characterized in that, The step of determining the clue information from the global information according to the task includes: Based on the correlation between the task and the global information, information with a correlation higher than a preset value is used as the clue information; And / or, The clue information is obtained by reasoning about the task using preset rules or a large language model.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Extract memory information from the first observed image to obtain local information; Update the global information based on the local information.

9. A navigation device, characterized in that, include: The acquisition module is used to acquire a first observation image of the intelligent agent, wherein the first observation image is a partial image of the scene captured by the intelligent agent through a shooting device; The determination module is used to determine clue information from global information based on the task input by the user. The global information is used to indicate the memory information stored by the agent. The memory information is used to describe the scene. The memory information is obtained based on a second observation image. The second observation image is taken by the agent when it visits the scene. The clue information is used to assist the agent in completing the task. The fusion module is used to fuse the clue information with the first observed image to obtain fused information; The reasoning module is used to reason about the fused information through a multimodal large language model to obtain action decisions, which are used to instruct the agent to perform the task.

10. The apparatus according to claim 9, characterized in that, The fusion module is specifically used for: The clue information is labeled onto the first observation image to obtain the fused information.

11. The apparatus according to claim 10, characterized in that, The fusion module is specifically used for: Based on the clue information, the position information of the clue information on the first observation image is calculated, and the position information is used to indicate the position of the clue information in the first observation image; Based on the location information, the clue information is labeled onto the first observation image to obtain the fused information.

12. The apparatus according to claim 10, characterized in that, The fusion module is specifically used for: Based on the clue information, a path is calculated, and the path is used to indicate the direction of the clue information relative to the agent; The path is projected onto the first observed image to obtain a fused image; Based on the clue information, the fused image is annotated to obtain the fused information.

13. The apparatus according to any one of claims 9 to 12, characterized in that, Before determining the clue information from global information based on the user-input task, the device further includes: The judgment module is used to determine whether the task has been completed based on the first observed image and obtain a judgment result.

14. The apparatus according to claim 13, characterized in that, The determining module is specifically used for: When the judgment result indicates that the task is not completed, the clue information is determined from the global information based on the task.

15. The apparatus according to claim 14, characterized in that, The determining module is specifically used for: Based on the correlation between the task and the global information, information with a correlation higher than a preset value is used as the clue information; And / or, The clue information is obtained by reasoning about the task using preset rules or a large language model.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The device further includes: The extraction module is used to extract memory information from the first observed image to obtain local information; An update module is used to update the global information based on the local information.

17. A navigation device, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method described in any one of claims 1 to 8 is performed.

18. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

19. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 8.

Images