850 results about "Semantic representation" patented technology

The invention discloses a multi-agent dynamic arrangement method based on multi-modal analysis and adaptive retrieval, and relates to the technical field of artificial intelligence and information retrieval. Comprising the steps of S1, converting a text, an image, structured data and voice content input by a user into a unified multi-mode semantic representation, S2, converting the unified multi-mode semantic representation into a specific execution process, and S3, automatically scheduling a reasoning agent, a knowledge obtaining agent and an execution agent according to DAG nodes, task elements and available resources, and obtaining the task elements and the execution agent according to the reasoning agent, the knowledge obtaining agent and the execution agent. S4, after task process construction and agent arrangement are completed, dynamic retrieval, evidence convergence and strategy optimization are carried out on information requirements related to a user task, so that a reasoning agent obtains complete knowledge support with consistent context, and S5, knowledge evidence is combined with a task process, so that the task process is completed. The method comprises the following steps: step S6, implementing problem solving, strategy generation and task closed-loop execution through a reasoning agent, step S6, performing actual operation on a target task by an execution agent according to an executable instruction sequence output by the reasoning agent, and outputting a result, and step S7, performing result verification according to an output result returned by the execution agent, and the correctness, integrity and consistency of an output result are examined through rule verification, model evaluation and evidence alignment.

The invention discloses a low-code development automatic generation method based on a large language model, which comprises the following steps: collecting natural language description information input by a user, and preprocessing; inputting the standardized demand corpus set into a large language model, and executing semantic understanding and context modeling; matching the low-code component library based on the structured semantic representation to generate component assembly description information; generating and verifying an engineering skeleton according to the assembly description information, and outputting an executable low-code application initial version; running the executable low-code application initial version, and monitoring and analyzing execution difference to generate an increment adjustment instruction; and inputting the increment adjustment instruction into the large language model, performing reconstruction and adaptive optimization, and outputting an executable low-code application final version. According to the method, large language model semantic understanding and adaptive optimization technologies are fused, automatic generation and continuous optimization of low-code applications are realized, and the method has the advantages of intelligence, high precision and engineering reliability.

The invention discloses a natural language-to-SQL (Structured Query Language) method and system based on RAG (Random Access Gateway) enhancement, and the method comprises the steps: constructing a query intention graph through dependency syntax analysis, recognizing and complementing semantic missing components, and forming complete semantic representation; vector representation is carried out on the business term segments by adopting vectorization coding, accurate definitions of business terms are obtained from a factory structured knowledge base, and an enhanced context set is formed through expansion retrieval in a low-confidence region; an SQL template mapping network is established based on historical query records, a mapping relation matrix is generated through field candidate matching, mapping conflict positions are identified, and multiple SQL candidate sequences are generated; grammar verification is carried out on the candidate sequence to identify grammar errors, semantic consistency verification is carried out to calculate the intention alignment degree, and an optimal SQL statement is selected through a deviation correction factor; and performing formatting processing and statistical abstract on an execution result, and providing data query capability for scenes such as factory quality management and equipment maintenance.

The invention provides a method and system based on NLP file analysis, and relates to the technical field of natural language processing. According to the method, time and identifier unification and format and character set standardization are carried out on the multi-source file, layout segmentation, table structure extraction, reference analysis, term standardization and anaphora resolution are combined, semantic representation is constructed, a hierarchical index and a unique traceability identifier are generated, intention recognition, retrieval sorting, incremental updating and consistency verification are supported, and the method is suitable for large-scale popularization and application. Unification, semantization and traceability of the file analysis process are achieved, and the processing efficiency and accuracy are improved.

The invention discloses an intelligent driving scene understanding and decision-making method and system based on a multi-modal large language model, and relates to the technical field of intelligent driving scene understanding and decision-making, and the method comprises the steps: collecting the visual, radar, laser radar, Internet of Vehicles, voice and vehicle state data of the surrounding environment of a vehicle, and forming a multi-modal original input set; performing feature extraction and semantic coding on various data in the multi-modal original input set to generate semantic feature vectors of corresponding modals; unified space mapping is carried out on the semantic feature vectors through a cross-modal alignment mechanism, multi-modal fusion processing is carried out based on an alignment result, and comprehensive semantic representation is generated; analyzing a social interaction relationship in the traffic scene based on the comprehensive semantic representation, identifying action modes and behavior tendencies of surrounding traffic participants, and generating social intention description information; and generating a scene query request according to the comprehensive semantic representation, and matching related traffic rules and driving experience in a pre-constructed driving common knowledge base.

The invention relates to the technical field of large model interaction, in particular to a long-term and short-term memory processing system based on a large model and a vector database, and provides a scheme that short-term context memory and long-term stable memory are subjected to hierarchical management and joint retrieval by performing semantic analysis and structured modeling on user interaction contents. Under the condition that historical interaction content exists, a theme segment division and intention stack mechanism is introduced, ordered organization and step-by-step alignment of intention states in multiple rounds of conversations are achieved, and matched memory entries are obtained from a cache database and a long-term database based on query semantic representation. Furthermore, by performing normalization, situational rewriting and conflict suppression on memory entries, controllable reasoning prompt information is constructed and input to a large model to generate target output, so that the consistency, stability and controllability of a generated result are improved in a multi-round interaction and cross-session scene.

The invention discloses a knowledge graph-based old building reconstruction scheme automatic recommendation method. The method comprises the following steps of S1, obtaining and preprocessing old building house data; s2, extracting a semantic entity and attribute relationship from a transformation case library, a building specification library and a construction scheme library, and constructing a knowledge graph; s3, mapping the house data to knowledge graph entity nodes, and executing graph embedding to generate building semantic representation; s4, constructing a graph neural network, calculating node semantic relevancy, and generating a building state vector; s5, inputting the building state vector into the reinforcement learning decision network, and optimizing the strategy to obtain an optimal transformation action; s6, screening reconstruction measures from the knowledge graph according to the optimal reconstruction action, and performing scoring to form candidate schemes; and S7, sorting the candidate schemes, selecting the scheme with the highest score, and recommending and updating the knowledge graph. According to the method, intelligent generation and self-optimization recommendation of the old building reconstruction scheme are realized, and the reconstruction efficiency of the scheme is remarkably improved.

The invention relates to the technical field of gift packaging three-dimensional modeling, and discloses a construction method of a gift box packaging design three-dimensional model. The method comprises the following steps: collecting natural language description, a freehand sketch image and a physical material attribute parameter; analyzing a natural language to construct a design intention map, and analyzing a sketch to generate a geometric feature code; performing knowledge alignment fusion on the two to form enhanced design semantic representation; generating an initial three-dimensional design framework by using a generative design algorithm; material parameters are integrated, compatibility simulation test is carried out, parameter self-adaptive adjustment circulation is started according to a feasibility evaluation report, and component parameters are dynamically optimized; and rendering and outputting a final model. Through deep fusion of multi-modal design information and preposed dynamic optimization of physical attributes, the accuracy of design intention expression and the producibility of a three-dimensional model are improved, and automation and intelligentization of a design process are realized.

The invention discloses a long document intelligent retrieval method and system based on hierarchical analysis and multi-modal fusion, and the method comprises the following steps: carrying out hierarchical analysis on an input long document, and obtaining a physical layout structure, a logic structure and a division and association relationship of multi-modal elements of the long document; respectively extracting text features of the text elements and visual features of the multi-modal elements; performing deep fusion on the text features and the visual features through a cross-modal fusion model to generate a unified multi-modal semantic representation vector; constructing a hybrid index; receiving user query, and performing preliminary retrieval by using the hybrid index to obtain a candidate result set; reordering the candidate result set; and outputting the retrieval result after reordering. Through visual-logic collaborative analysis, gating enhanced bilinear attention fusion and graph attention-based context reordering, a collaborative gain effect is generated in a long document intelligent retrieval task, and the retrieval precision and the user experience are remarkably improved.

The invention discloses an electric power AI security detection model optimization method and system fusing attribution quantification and adversarial correction. The optimization method comprises the following steps: step 1, carrying out structured semantic representation on heterogeneous security alarms of an electric power network; 2, performing model decision logic analysis based on hybrid attribution quantization; step 3, automatically diagnosing decision prejudice based on domain knowledge masks; step 4, constructing an adversarial sample generated based on an anti-fact text; and 5, performing closed-loop fine adjustment and optimization on the attribution regularization model. According to the method, the interpretable ability of large model decision analysis, the root cause positioning ability of misinformation and the autonomous repair optimization ability are improved, the transparency and credibility of model decision are improved, the model misinformation caused by environmental influence is reduced, and the efficiency of model autonomous correction is improved.

The invention discloses an incomplete multi-view multi-label data classification method based on semantic enhancement and pseudo-label uncertainty perception, and the method comprises the steps: employing a dual-channel feature extraction and decoupling module to obtain the shared semantic representation and specific representation of each view in each sample for a constructed incomplete multi-view multi-label data classification network model; performing cross-view fusion on the shared semantic characterization and the specific characterization, obtaining a unified shared characterization and a unified specific characterization corresponding to each sample, performing feature fusion, obtaining a fusion characterization of each sample, inputting the fusion characterization of the sample output by the dual-channel feature extraction and decoupling module into a classifier for multi-label prediction, and performing multi-label prediction on the fusion characterization of the sample. Therefore, a multi-label classification prediction result is obtained, and model training is carried out based on a total contrast learning loss function and a joint supervision classification loss function. According to the method, the classification performance and the model robustness on incomplete multi-view multi-label data are remarkably improved through training learning under the guidance of semantic enhancement and uncertainty.

The invention discloses an entity relationship recognition method based on rotation position coding and a global pointer network, which comprises the following steps of: coding an input Chinese threat intelligence text by utilizing a pre-training language model fused with the rotation position coding, and generating context semantic representation with enhanced position perception capability; based on the context semantic representation, decoding all possible entity spans and types thereof in parallel in a two-dimensional grid space through a global pointer network to generate an entity set; for a target entity pair in the entity set, constructing a structured input sequence containing entity position information; processing the structured input sequence by using a double-attention coding mechanism; and based on the output of the double-attention coding mechanism, determining the relationship type between the target entity pairs through a relationship classification module. According to the method, precise decoding of nested entities and robust recognition of cross-language terms are realized through geometric space mapping and a dynamic boundary optimization mechanism.

The invention discloses a private network content copyright monitoring and evidence obtaining system based on AI and a large model, which utilizes AI and large model technologies to carry out copyright monitoring and evidence obtaining on multimedia content in a private network environment, and carries out deep semantic understanding and cross-modal feature extraction through a large model processor to generate unified semantic representation. The method comprises the steps that a copyright content database is established and stored in a copyright content knowledge base, the copyright content knowledge base is used for storing metadata of original content protected by copyright, unified semantic representation and copyright declarations in a natural language form provided by a copyright party, and the copyright declarations are converted into query vectors through a semantic understanding technology; the method comprises the following steps: acquiring a semantic representation of a multimedia content, performing similarity calculation with the semantic representation of the multimedia content, identifying infringement content, when the infringement content is identified, recording an original source, publishing time and publisher information of the infringement content, performing differentiation analysis, generating an evidence chain, displaying a copyright monitoring result through a user interface, generating infringement alarm information, and presenting details of the evidence chain.

The invention provides a multi-modal knowledge graph construction method and system in the agricultural field, and relates to the technical field of data processing. The method comprises the following steps: performing recognition and structured expression on agricultural entities, relationships and events by fusing texts, images and time-space data to form an agricultural semantic representation model; calculating a confidence coefficient based on the evidence template and resolving conflicts, and generating a structured knowledge unit; an agricultural multi-modal knowledge graph is constructed on the basis of the event network, and dynamic updating is achieved through increment correction and self-correction; and finally, outputting an agricultural aid decision result with confidence by combining causal reasoning and confidence evaluation. The agricultural multi-modal knowledge graph construction method solves the problem that an agricultural multi-modal knowledge graph construction method in the prior art lacks a system modeling and closed-loop optimization mechanism for cross-modal alignment credibility, ontology constraint, evidence tracing, causal verification and dynamic updating.

The invention discloses an equipment agent processing program conversion method and device based on cooperation of a large model and a small model. An equipment agent is constructed by analyzing control systems, kinematics structures and process constraint information of a source machine tool and a target machine tool, and line-level instructions are abstracted into system-independent unified semantic representation based on a multi-system numerical control corpus. And performing grammar analysis and semantic mapping on the source program, and generating a conversion task in combination with grammar rules, kinematics accessibility and process security constraints of a target machine tool. A small model is finely adjusted through supervised learning to realize basic generation capability, a large model is introduced as a patch evaluator, candidate results are jointly scored from different dimensions, knowledge migration is realized based on improved near-end strategy optimization, and a small model generation strategy is continuously optimized. And finally, high-quality results are screened through confidence degree sorting, cross-system G / M instruction and multi-axis track generation and verification are completed, and high-precision automatic conversion of numerical control programs is achieved.

The invention discloses an automatic causal structure generation method based on semantic representation and logical reasoning of a large language model. The method comprises the following steps: acquiring an input text; performing semantic coding and clustering on the obtained input text by utilizing a large language model, and establishing a candidate causal variable set; causal relationship detection is carried out on the established candidate causal variable set based on anti-fact intervention and do-calculation; performing causal direction judgment, and generating a directed acyclic causal graph meeting logic consistency; and on the basis of the generated directed acyclic causal graph, natural language interpretation is generated by using a large language model, and logic consistency closed-loop verification is carried out. According to the method, automatic generation from the natural language to the causal structure is realized, the causal variable set is automatically extracted and constructed from the unstructured natural language text, the defects that variables need to be manually defined and modeling depends on field experts in the existing causal modeling process are avoided, and the labor cost and professional threshold of causal structure construction are remarkably reduced.

The invention discloses a multi-modal data fusion method in electric power overhaul, which comprises the following steps of: firstly, respectively acquiring long-sequence video data and audio data in an electric power overhaul process, and constructing a window-level multi-modal data fusion model; secondly, performing time window division and coarse alignment on the long-sequence video data and audio data by the window-level multi-modal data fusion model, generating a frame-level quality score, and performing fine alignment on multi-modal characteristic data in a time window by adopting a segmented time sequence alignment and compensation algorithm; and calculating a window-level quality score, a window-level weak supervision label and a confidence coefficient, and finally performing multi-modal fusion according to the multi-modal feature data after fine alignment and the window-level weak supervision label to generate a window-level semantic representation as a model output. According to the method, accurate time alignment of video and audio data and consistent fusion of multi-modal semantic features are realized, and more reliable alignment data and semantic input are provided for subsequent target recognition.

The invention discloses a hierarchical semantic enhanced large model zero sample learning method, which is characterized by comprising the following steps: firstly, performing semantic enhancement on category names by using a large language model to generate rich text description, and constructing a dynamic and hierarchical semantic prototype by combining original semantic information, the prototype comprising global concepts, local attributes and relation representations; secondly, extracting global semantic features and local detail features of the image by adopting a vision-language large model and convolutional neural network double-branch structure, and fusing the global semantic features and the local detail features through an attention mechanism to obtain enhanced visual representation; finally, multi-level global alignment, local alignment and relation alignment are designed and jointly optimized, accurate mapping of enhanced visual features and hierarchical semantic prototypes is achieved on multiple granularities, and classification of invisible categories is finally completed. The objective of the invention is to solve the problem of limited model generalization ability caused by insufficient semantic representation and single vision-semantic granularity in the existing zero sample learning method, and to enhance semantic representation by introducing knowledge of a large language model and innovatively implement hierarchical alignment, so that the robustness of the zero sample learning method is improved. And the recognition precision and robustness of the model in traditional and generalized zero sample learning scenes are remarkably improved.

The invention relates to a semantic collaboration modeling method and system for multi-modal sequence recommendation, belongs to the technical field of multi-modal sequence recommendation, and aims to solve the problems that an existing method depends on an article ID, multi-modal semantic collaboration signals are difficult to mine, the cross-scene migration capability is weak, and the representation precision is insufficient. The method comprises the steps that text information and image information of an article are extracted according to historical interaction behaviors of a user, and multi-level semantic representation is extracted through a multi-mode encoder; a multi-head attention module with a one-way mask is adopted to capture semantic collaboration signals, and the semantic collaboration signals are integrated into initial modal representation; performing fine-grained semantic focusing and optimization by using a hybrid expert structure; obtaining a final modal representation of the article through a hybrid expert fusion module; and constructing a multi-modal behavior sequence of the user based on the final modal representation of the article, and calculating preference scores after coding by a sequence recommendation model to realize next-step interaction prediction of the user. According to the method, recommendation accuracy, generalization ability and cross-scene knowledge migration efficiency can be remarkably improved.

The invention relates to the technical field of data processing, in particular to a relational database intelligent query method based on machine learning, which comprises the following steps of: processing multi-modal flow data through time sequence alignment, generating a unified semantic representation vector, constructing a dynamic psychological state map, and modeling a psychological state evolution track by utilizing a neural common differential equation mechanism. After user query is received, historical dialogue nodes are retrieved from the graph, enhanced query intention representation is generated, the enhanced query intention representation is converted into an execution plan through a neural symbol inference engine, and a graph neural network is adopted to predict execution cost. And finally, a personalized analysis report is generated by combining a causal discovery algorithm, and system adaptive optimization is realized through feedback signals. According to the method, the problems of inconsistent time sequence semantics and strong context dependency of the multi-modal psychological data are effectively solved, and the query accuracy and the personalized level in a psychological dialogue scene are improved.

The embodiment of the invention provides a context construction method and device for an agent and a storage medium, and the method comprises the steps: obtaining the current input information of the agent in one interaction round, and carrying out the analysis to generate a current semantic representation; determining a semantic association degree between the current semantic representation and the context of the current dialogue task, and calculating a long-term value score of the current semantic representation; under the condition that the semantic association degree is greater than a first preset threshold value, storing the current semantic representation into a short-term memory library; under the condition that the long-term value score is greater than a second preset threshold value, storing the current semantic representation into a long-term memory library; when the intelligent agent needs to generate a response for the current input information, searching target memory content related to the semantics of the current input information from the short-term memory library and the long-term memory library; and combining the target memory content with the current input information to form prompt information, and inputting the prompt information into a large language model to generate a response for the current input information.

The invention provides a file positioning management method and system based on artificial intelligence, and relates to the technical field of artificial intelligence. According to the method, files are collected from multiple sources and subjected to standardization processing, an element set is generated in combination with multi-modal analysis of texts, images, audios, videos and tables, cross-modal alignment is achieved through a semantic representation model, hierarchical indexes of semantics, keywords and relations are constructed, and a unique traceability identifier is generated; in the query stage, intention recognition and joint retrieval are carried out, a result subjected to permission verification and traceability information labeling is output, online optimization and incremental reconstruction are executed based on user feedback, and comprehensiveness, accuracy, traceability and self-adaptive optimization of file positioning are achieved.

The invention discloses an electric power field intelligent portrait generation method and device based on a multi-modal knowledge graph, and belongs to the technical field of artificial intelligence. The method comprises the steps that a knowledge representation framework is adopted to perform standardized semantic representation on entities and relationships extracted based on electric power field multi-source heterogeneous data, and a graph database is adopted as a storage platform; constructing a multi-modal electric power knowledge graph; metadata extraction and cross-modal semantic alignment are carried out based on the multi-source heterogeneous data, and cross-modal feature fusion is carried out on the multi-modal metadata set based on the multi-modal electric power knowledge graph to obtain a plurality of multi-modal feature vectors; constructing a multi-modal graph neural network based on the multi-modal electric power knowledge graph, the multi-modal metadata set and each multi-modal feature vector; and performing multi-modal portrait fusion based on each node feature of the multi-modal graph neural network to generate a multi-modal intelligent portrait. Therefore, by implementing the method and the device, the intelligent portrait fused with the multi-dimensional power metadata can be generated.

The invention provides an intelligent document understanding method and system combining natural language processing and deep learning, and relates to the technical field of natural language processing and information management. Fusing the semantic pre-annotation result and the original text feature of the document, inputting the fused semantic pre-annotation result and the original text feature of the document into a cross-modal semantic enhancement model to obtain a document enhanced semantic representation, performing bidirectional semantic interaction with a dynamic business knowledge network based on the document enhanced semantic representation, generating an associated interaction result, and constructing a structured semantic asset; finally, the structured semantic assets are input into an intelligent document application engine, semantic feedback data are collected to optimize model parameters, document understanding accuracy and practicability can be improved, and diversified business requirements are met.

The invention belongs to the technical field of intelligent and robot operation, and discloses a collaborative robot closed-loop operation method based on scene perception and thinking chain reasoning, which comprises the following steps of: firstly, generating structured semantic representation containing confidence by utilizing a visual language model of LoRA efficient fine tuning and fusing multi-view data, and providing semantic understanding for a robot; secondly, utilizing a three-dimensional diffusion model to generate explicit geometric priori under the condition of no real CAD, and correspondingly completing PnP initial pose estimation in combination with 2D-3D; then, carrying out pose optimization by adopting a tracking-refining strategy, and triggering reinitialization based on geometric prior when a residual error exceeds a threshold value; and finally, decomposing a global task by utilizing a thinking chain reasoning module, and realizing'perception-reasoning-execution 'closed-loop control based on visual servo in cooperation with an execution module. The method realizes effective combination of large model reasoning and physical execution, has strong robustness under dynamic disturbance, and is suitable for intelligent manufacturing and man-machine cooperation scenes.

The invention discloses a multi-modal information fused high-myopia retina image classification method and system. The method comprises the following steps: acquiring an OCT image of a to-be-classified high-myopia retina and structured numerical data corresponding to the image; the OCT image is preprocessed, the preprocessed image is input into an image encoder for high-dimensional semantic representation learning, and overall semantic vector representation of the image is obtained; converting the structured numerical data into a medical language description text, and inputting the medical language description text into a text encoder for deep semantic modeling to obtain overall semantic vector representation of the text; inputting to a multi-modal fusion module, carrying out feature interaction and fusion through a bidirectional cross attention mechanism, and generating a fused multi-modal feature; and outputting a retina splitting stage category corresponding to the OCT image through a classification module. According to the method, the OCT image, the structured numerical data and the split staging definition text are utilized to perform multi-modal feature fusion, so that the accuracy of image staging recognition is improved.

The invention discloses an archive content semantic clustering method based on a graph neural network, and the method comprises the following steps: S1, carrying out the word segmentation, denoising and vector expression of an archive text, and generating a text feature vector set; s2, constructing a semantic graph structure model according to the semantic similarity and the reference relationship between the texts; s3, generating a division result with a balanced structure on the semantic graph by adopting a graph division algorithm; s4, performing double-layer node merging on each graph division cluster, and generating a graph structure coarsening result and a mapping relation; s5, inputting the original graph and the coarsened graph into the graph neural network model, and calculating and fusing each layer of semantic representation; s6, cross-layer consistency constraint optimization node semantic representation is introduced, and a unified embedded vector set is generated; and S7, inputting the embedded vector into the clustering model, and outputting a semantic clustering category of the archive text. According to the invention, semantic recognition and automatic grouping of archive contents are realized.

The invention discloses a cross-border digital service multi-language real-time interaction and semantic error correction method and system, and belongs to the technical field of text processing.The method specifically comprises the steps that when cross-border interaction begins, voice, text and auxiliary multi-mode information of a user are collected, fused semantic representation is generated, the semantic representation is input into a cross-language prediction model, and the cross-language prediction model is obtained; a target language candidate result is obtained, a reverse mapping channel is established, when semantic errors or ambiguity occurs in the target language candidate result, collaborative correction is conducted in combination with the scene rule base, user historical preferences and real-time feedback, the corrected result is processed through a predefined target context sensitive model, and the target language candidate result is obtained. Automatically adjusting the culture expression, the terminology and the compliance, and outputting to the cross-border service terminal in real time; according to the method, the target context sensitive model is introduced in the output stage, cultural expression, terminologies and law compliance are automatically adjusted, and multi-language seamless communication and high-reliability semantic transfer in cross-border digital services are achieved.

The invention relates to the technical field of software engineering and information processing, in particular to a semantic verification method and device based on a large model.The method comprises the steps that a source code file and a corresponding software requirement and design document are obtained, code semantic representation is generated based on grammar analysis and static program analysis, and a semantic verification result is obtained; based on text analysis and business flow chart structure identification, generating a design demand semantic representation; respectively inputting the code semantic representation and the design demand semantic representation into a pre-training language large model to generate a code implementation logic description and a design demand logic description; and performing semantic consistency verification on the two types of logic descriptions to obtain a semantic inconsistency result, determining a code position in the source code file based on the semantic inconsistency result, and outputting a scanning report. According to the invention, the automation degree and the rechecking performance of consistency checking of requirements can be improved, and the leak detection and reworking cost can be reduced.

The invention discloses an interpretable end-to-end automatic driving method based on a multi-modal large model, and belongs to the technical field of data processing, and the method comprises the following steps: S1, obtaining a driving environment image, carrying out the feature extraction and coding, and generating a visual representation; s2, generating scene semantic representation by using the semantic answer; s3, determining a state by using a finite state machine; s4, generating a final action according to the visual representation and the scene semantic representation, and constraining the final action through the state of the finite-state machine; and S5, based on the final action, generating a language instruction. According to the method, navigation information and human natural language instructions are encoded into prompts, and large model behavior generation is guided.