6444 results about "Speech model" patented technology

The invention discloses a big language model reasoning optimization method and system based on dynamic video memory compression and memory isomerism, and intelligent management of video memory resources is realized by integrating a dynamic compression strategy of KV Cache and a memory parallel architecture. The method comprises the following steps: 1) analyzing the spatial-temporal characteristics of the KV Cache in real time, adaptively selecting a quantization compression algorithm, a rarefaction algorithm or a low-rank decomposition algorithm, performing hierarchical storage based on attention head importance scores, keeping high precision of a core head, and implementing low-bit quantization on a secondary head; (2) the compressed inactive data are divided into a plurality of data blocks to be stored in a system memory, a parallel data channel group is established according to the number of physical channels, the compressed blocks are concurrently read through multiple channels during loading, and parallel decompression of a sparse matrix is accelerated through a GPU tensor core; and 3) constructing a KV Cache multiplexing mechanism and a parallel channel, and parallelizing a compression / decompression process and model calculation by adopting a hardware acceleration compression and asynchronous pipeline mechanism.

The invention belongs to the technical field of artificial intelligence and big data, and discloses a customer data processing and insight system based on a big language model. The system is composed of a multi-source data access module, a data preprocessing and label fusion module, a large language model semantic understanding module, a knowledge enhancement and semantic linkage module, an insight generation and visualization module, an intelligent strategy output module and a feedback learning and self-optimization module. According to the method, multi-source heterogeneous data such as texts, voices and structured behaviors are integrated, and the deep semantic analysis capability of a large language model is combined, so that global modeling of customer behaviors and intentions is realized; a multi-modal synchronous acquisition and standardization mechanism eliminates data format barriers, and a dynamic label mechanism adapts to context changes, so that the system can capture deep semantic association in customer expression, and compared with a traditional keyword matching method, the semantic understanding accuracy is improved by more than 40%, and a more complete data base is provided for insight generation.

The invention relates to the technical field of voice semantics, can be applied to business scenes of financial science and technology, medical health, poster design and the like, and discloses a visual sense generation method and device based on semantic association modeling, equipment and a medium. Generating a demand text containing theme and style parameters; semantic features in the demand text are extracted, semantic association weights are constructed, and element layout coordinates are optimized in combination with spatial distribution constraints; and encoding the layout information into a control matrix, fusing the control matrix with the initial noise, adjusting a noise reduction process through an encoding and decoding network, and generating target visual content highly matched with the semantic meaning of the user instruction. According to the method, the layout optimization function is constructed, the diffusion model is guided to focus the semantic salient region in space, language model output and the visual generation process are closely combined, structured response and space mapping of user semantic requirements are achieved, and the expression consistency and personalized adaptation capacity of visual content generation are improved.

The invention relates to the technical field of artificial intelligence, and discloses a bidding document generation method based on retrieval enhancement generation and a large language model, and the method comprises the steps: analyzing technical parameters, legal terms and score weights in a bidding demand document; retrieving matched historical bidding document fragments and technical specifications from the industry knowledge base, and generating a retrieval enhanced data set; fusing the data through a dynamic weight distribution algorithm and generating an initial bidding document draft; checking conflict terms based on the legal semantic knowledge graph and marking correction suggestions; and optimizing the bidding document structure and the key content according to the score weight. According to the method, an online learning mechanism driven by sentence vector matching, timeliness weight calculation and user feedback is adopted, so that the problems of inaccurate technical parameter extraction, low legal conflict detection efficiency and non-optimized scoring rules in the bidding document generation process are solved.

The invention discloses an archive knowledge base construction and retrieval method and system based on multi-modal data fusion. The method comprises the steps that heterogeneous archive data are cleaned, image features are extracted through CNN, text features are extracted through Transform, audio is converted into text and then subjected to similarity, a unified feature vector is generated, and metadata is constructed according to archive code association; creating a graph database instance, defining nodes and relationship types, importing entities and relationships, and storing feature vectors and metadata; the features are mapped to a high-dimensional shared semantic space, positive and negative sample pairs are constructed to update embedded layer parameters, self-attention is used in modalities, a shared attention mechanism is used between modalities, weights are adjusted according to archive features, and unified knowledge representation is generated; segmenting the steering quantity of the multi-modal data, storing the steering quantity into a database, and adopting hierarchical indexing and optimizing as required; related document fragments are retrieved through RAG technology vectors, answers are generated with the help of a large language model, and session feedback is provided. The file retrieval efficiency and accuracy are improved.

The invention discloses a power grid equipment intelligent operation and maintenance method based on a large language model and a knowledge graph, and relates to the technical field of intelligent power grid operation and maintenance, and the method comprises the steps: carrying out the operation and maintenance of power grid equipment through a power grid operation and maintenance knowledge graph constructed through a large language model and a knowledge federation technology, the power grid equipment operation and maintenance comprises one or more of health state evaluation, fault risk prediction and early warning, intelligent operation and maintenance strategy generation and health degree dialogue query of the power grid equipment. According to the invention, a power grid operation and maintenance mode can be effectively promoted to be transformed and upgraded from a traditional manual experience type and a passive maintenance type to a data-driven, intelligent and active preventive maintenance mode. Key intelligent operation and maintenance technical support is provided for building a novel electric power system with new energy as a main body, the novel electric power system is assisted to achieve the development goals of being safer, more efficient, cleaner and lower in carbon, and important industry strategic significance and social contribution are achieved.

The present invention provides a large language model-based system and method for data processing in application developed by codeless platform. The invention includes identification of intent of a user to process procurement, supply chain, application integration, application restructuring or development scenarios.

The invention relates to the technical field of intelligent operation and maintenance and artificial intelligence crossing of a power system, in particular to an intelligent power work order processing method of an RPA fused multi-modal large model, which analyzes multi-modal work order data such as texts, voices, images and the like through a domain adaptation large language model, and realizes fault key information extraction and conflict resolution in combination with a dynamic knowledge graph; performing work order priority scoring and resource allocation by using space-time constraint reinforcement learning; an analysis result is converted into an automatic execution script through an RPA engine, and a whole-process closed loop of order sending, processing and feedback is achieved; meanwhile, a feedback optimization and conflict resolution cooperation mechanism is constructed, and the knowledge graph and the model precision are continuously iterated. The method improves work order processing efficiency and analysis precision, enhances decision scientificity, and is suitable for an intelligent operation and maintenance scene of a power system.

The invention relates to the technical field of intelligent conference processing, and relates to a conference summary processing method and system using AI, and the method comprises the steps: carrying out the real-time noise suppression of a collected conference audio stream and associated text data through a noise suppression algorithm, and carrying out the cross-modal alignment of the denoised data through a cross-modal alignment algorithm; a domain-specific attention head is inserted into an attention layer of the pre-trained Transform model, a domain-enhanced speech recognition model is constructed, and audio is converted into a text sequence with a speaker tag; adopting a heterogeneous graph neural network to construct a structured topic evolution graph; key decision nodes in the structured topic evolution graph are extracted based on a reinforcement learning strategy, and a final conference summary document is generated. In the decoding stage, the fusion proportion of the acoustic model and the language model is dynamically adjusted based on the real-time acoustic confidence coefficient, the recognition rate of the vocabularies in the professional field is increased, and the problems of frequent term transcription errors and poor semantic coherence in the professional conference are effectively solved.

The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating personal responses through retrieval-augmented generation. In particular, the disclosed systems can generate a query embedding from a query generated by an entity and determine data context specific to the entity by comparing the query embedding with a plurality of vectorized segments of content items associated with the entity. The disclosed systems can provide the data context to a large language model and generate a personalized response informed by the data context. Subsequently, the disclosed systems can provide the personalized response for display on a client device associated with the entity.

The invention discloses a multi-modal interview automatic quality analysis and evaluation method and system based on a large model, and the method comprises the steps: collecting and storing multi-modal data, such as texts, audios, videos and behavior interaction, and carrying out the preprocessing of the multi-modal data to form a standardized data set; utilizing a preset interview structure and a large model to dynamically guide the process, adjusting the topic rhythm according to real-time feedback, and recording stage conversion information to form logic trajectory data for process coherence management; automatically coding text data through a large language model, extracting features such as keywords and performing topic clustering, performing cross validation and semantic fusion in combination with data analysis results of each modal, and generating deep analysis results such as psychological states; and generating a comprehensive assessment report containing qualitative description, quantitative score and psychological abnormality or cognitive disorder risk prompts based on a deep analysis result, thereby providing a basis for psychological health assessment and cognitive competence evaluation. According to the method, automatic analysis of multi-modal data is realized, and evaluation scientificity and efficiency are improved.

The invention relates to the technical field of large language models, and discloses a voice data interaction feedback control processing method based on a large language model. The method comprises the following steps: acquiring an industrial voice instruction through an AMR main controller, and obtaining a standardized vector through industrial lexicon matching and intention classification; inputting an industrial large language model for reasoning processing, and generating an AMR execution scheme; carrying out distributed coordination and task allocation on the AMR cluster to form a control instruction sequence; and the motion controller executes monitoring, processes exceptions and outputs a feedback strategy. Accurate classification and semantic understanding of complex industrial instructions are achieved, the processing capacity of professional knowledge in the industrial field is improved, and meanwhile high-precision real-time state monitoring is achieved.

The invention relates to the technical field of data processing, in particular to a method and system for processing data based on a large language model.The method comprises the following steps that based on inter-sentence punctuation positioning and part-of-speech tagging, sentence blocks are divided to generate functional partitions, themes and relational words are extracted to judge semantic chain starting points, a trigger index sequence is constructed, and logic jump points and breakpoint positions are recognized; and mapping the label structure to the language model output analysis deviation, and generating a label mapping combination list. According to the method, semantic turning nodes can be captured by analyzing the semantic direction change trend, the relation between the semantic turning nodes and verb and noun combinations is judged, the position of a trigger point of an actual information transfer effect is extracted, and the break point area of a semantic path is recognized through the positioning of key word starting and stopping blocks and the logical judgment of noun group cross combination; the path integrity has a clear fracture identifier, a traceable semantic mapping structure path is established in a language model, and the accuracy, coherence and hierarchy clearness of semantic reconstruction are enhanced.

A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments.

The invention provides an industrial agent decision-making method and device, electronic equipment and a storage medium. The method comprises the following steps: loading multi-source industrial data to a knowledge graph, and constructing a domain ontology and an MCP ontology; performing parameter fine tuning on a preset base large language model; receiving an industrial process planning demand input by a user, and performing semantic analysis on the industrial process planning demand; processing the query result by utilizing an MCP context strategy to generate a prompt context; inputting the prompt context and the industrial process planning demand into a fine tuning large language model to generate a first process planning scheme; performing rule verification on the first process planning scheme based on the knowledge graph, and judging whether the first process planning scheme meets a preset rule or not according to a verification result; and converting the final process planning scheme into a control script or an interface calling sequence which can be analyzed by an industrial execution system, and outputting the control script or the interface calling sequence. According to the method, the cross-software system compliance process planning with unified knowledge, automatic generation and rule verification can be realized.

A computer-implemented method for jailbreaking Large Language Models (LLMs) requiring only black-box access-Branch-and-Prune. LLMs are powerful tools displaying many capabilities. However, despite safety training, LLMs can generate harmful, biased, and toxic content-demonstrated by the prevalence of human-designed “jailbreaks” that override LLM's safety “guardrails.” At the core, the method engages an “attacker” LLM into conversations where it generates variations of the original prompt that may jailbreak the target LLM. Compared to prior methods, the disclosed Branch-and-Prune adaptively decides which conversations to engage in (pursuing conversations with a high-likelihood of success while abandoning less-promising ones), and “mixes” the different conversations (increasing the success rate of the method). This enables jailbreaks for state-of-the-art LLMs for over 80% of the existing harmful prompts while requiring fewer queries to the target LLM.

The invention discloses a city updating intelligent expert system architecture and method based on a large language model, and relates to the technical field of city planning and construction. The knowledge base management module supports efficient retrieval and application of professional domain knowledge; the expert agent group comprises a user-defined core expert and a system dynamic supplement expert; the expert team collaborative decision-making module is used for executing multiple rounds of hierarchical collaborative decision-making processes; the meta-agent module is responsible for project feature analysis and expert team dynamic configuration; the auxiliary agent module comprises a host agent which is responsible for guiding the decision making process; the recorder agent generates a decision summary report; the verification agent is responsible for verifying and evaluating the professional ability of the expert agent; and the scoring agent evaluates the decision quality. According to the method, the problems of dispersed professional knowledge, low cross-domain expert cooperation efficiency, high decision-making cost and the like in the existing city updating decision-making process are solved, and the decision-making efficiency and quality of city updating early-stage planning are comprehensively improved.

The invention discloses a large-model real-time voice text intention recognition method and system based on text prediction, and the method comprises the steps: obtaining the real-time voice data of a user, carrying out the real-time voice recognition processing through a streaming voice recognition interface, and obtaining a part of transcriptional text; inputting the partial transcription text into a mask language model for text prediction, and generating a plurality of high-credibility complete sentence candidates; based on the complete sentence candidates, the complete sentence candidates are input into a large language model in parallel for intention recognition, a corresponding intention result is obtained, and a mapping relation between the candidate sentences and the intention recognition result is established; and obtaining a sentence completely expressed by the user, calculating the similarity between the complete actual sentence and a plurality of high-credibility complete sentence candidates through a multi-level text similarity algorithm, selecting the candidate sentence with the highest similarity score, and directly obtaining a corresponding final intention recognition result based on the mapping relationship. The objective of the invention is to solve the technical problem of high response delay of an existing voice intention recognition system.

The invention discloses a robot anthropomorphic interaction method based on multi-modal emotion recognition and customized portrait generation. The method comprises the following steps: S1, dynamically fusing multi-modal emotions; the method comprises the following steps: S1, synchronously acquiring voice, visual and text signals through a multi-source heterogeneous sensor, capturing a user voice stream by a high-fidelity microphone array, and extracting acoustic characteristics such as intonation and speed, S2, performing cross-modal reasoning; s3, synchronously generating contents; step S4: style migration; step S5, anthropomorphic voice and expression generation; according to the method, man-machine interaction emotion is analyzed and generated by utilizing a large language model and multi-modal information fusion, the singleness of interaction emotion and the deficiency of emotional sharing ability are avoided, a strong emotion interaction characteristic is achieved, the image of the robot is obtained through a generative technology and can be migrated to any image, the limitation that a specific image is independently made is broken through, and the interaction effect of the robot is improved. The advantage that one robot can be suitable for different scenes is achieved.

A large language model predicts a topic summarization of a long document given a set of segments from the long document that pertain to a topic of interest. The set of segments from the long document are selected by searching for similar segments from other documents that have been labeled to indicate whether or not the segment pertains to a topic of interest. The search is based on an embedding of a segment from the long document closely matching embeddings of the labeled segments. Each segment of the long document is scored based on the labels of the closest-matching similar segments. The segments from the long document are ranked by their respective score and the highest-scored segments are included in a prompt to the large language model for the model to generate a topic summarization of the long document.

The invention relates to a metro field large language model evaluation method and system. The method comprises the following steps: step 1, constructing a data set; the step 1 comprises the following steps of: 1.1, collecting public professional data; step 1.2, collecting internal professional data; 1.3, collecting a general data set; 2, data processing; in the step 2, the following steps are executed: step 2.1, data formatting; step 2.2, data cleaning is carried out; step 2.3, performing data enhancement; 3, fine adjustment of the model; in the step 3, the following steps are executed: step 3.1: constructing a training data set; step 3.2, fine adjustment of the model; step 3.3, carrying out model evaluation; 4, constructing an evaluation system; in the step 4, the following steps are executed: step 4.1, determining evaluation indexes; step 4.2, constructing an evaluation system; step 4.3, constructing an intelligent agent; step 4.4, multi-agent cooperation is carried out; and 5, performing model evaluation.

A computing platform may generate, using a test case generation model, a plurality of large language model (LLM) test cases. The computing platform may input, into an LLM, the plurality of LLM test cases, which may produce a plurality of unverified LLM test results. The computing platform may input, into a validation model, the plurality of LLM test cases, which may produce a plurality of validated LLM test results. The computing platform may compare, using a falsified output evaluation model, the plurality of unverified LLM test results with the corresponding plurality of validated LLM test results, which may produce an LLM compliance score for the LLM. The computing platform may compare the LLM compliance score to a compliance threshold. Based on identifying that the LLM compliance score meets or exceeds the compliance threshold, the computing platform may automatically deploy the LLM for use in an enterprise environment.

Techniques for leveraging a large language model (LLM) in software development are described. A description of an error message associated with a software system hosted by the multi-tenant provider network is received. An LLM is prompted with an analysis prompt to generate an analysis of the error message, the analysis prompt including the description of the error message, and the analysis of the error message is received from the LLM. The LLM is prompted with a suggested resolution prompt to generate a suggested resolution to a cause of the error message, the suggested resolution prompt including the description of the error message and the analysis of the error message, and the suggested resolution is received from the LLM. A change to resolve the cause of the error message is sent to an originator of the received description, the change based at least in part on the suggested resolution.

The invention relates to the technical field of large models, and discloses a knowledge graph and retrieval enhancement-based government affair question-answering method and system, and the method comprises the steps: carrying out the information extraction of each government affair document, and obtaining text information, multi-modal information and metadata; fusing the information to obtain document information; performing entity relationship extraction based on the document information to obtain entity, relationship and text key value pairs; constructing a government affair knowledge graph based on the entity, the relationship and the text key value pair corresponding to each government affair document; based on the type of the user query statement and the government affair knowledge graph, adopting a retrieval enhancement technology to generate a query answer; and optimizing the query answer to generate a target answer. According to the method, through the multi-modal information and the knowledge graph, the document content and the query intention are comprehensively understood, the question and answer accuracy is improved, the knowledge graph and a retrieval enhancement technology are fused, the illusion phenomenon of a large language model is reduced, the answer accuracy is improved, the safety, accuracy and readability of the answer are ensured through optimization, and the user experience is improved.

The invention discloses an AI generation content detection and review method, device and equipment and a storage medium, and the method comprises the steps: receiving multi-modal input data which comprises text data, image data and video data; calling a large language model to carry out compliance analysis on the text data to obtain an analysis result; when the analysis result is compliance, calling a multi-modal model to identify and detect the image data and the video data, and respectively obtaining an image detection result and a video detection result; and aggregating the analysis result, the image detection result and the video detection result to obtain a content detection result. According to the method, the processing paths are automatically allocated according to the content types, redundant calculation is avoided, hardware consumption is remarkably reduced, the multi-modal detection results are aggregated into structured output, the large-batch content processing efficiency is remarkably improved, calculation resource occupation is greatly reduced, and seamless integration of the detection results and a downstream service system is achieved.

The invention relates to the technical field of robot control, and discloses a robot control method, system, equipment and medium based on a multi-modal large model, and the method comprises the steps: collecting the multi-source modal data of a scene where an operation task is located, and carrying out the processing through a machine learning model, obtaining a multi-modal feature, and carrying out the position coding and Transform fusion processing, multi-modal fusion features are obtained, the multi-modal fusion features and the constructed job task knowledge base are input into a large language model to decompose a target job task, a human-in-the-loop mechanism is introduced to optimize a decomposition result, and a sub-task sequence is obtained; according to a subtask type in the subtask sequence, processing the subtask sequence through a visual language action model or a reinforcement learning model, and generating a motion instruction to enable the robot to start an execution process of the target operation task; live-line work tasks are processed through the multi-modal large models LLM, VLA and the like, and the work efficiency of the autonomous distribution network live-line work robot is improved.

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

A computer-implemented method for operating an artificial intelligence voice agent system includes receiving voice input through communication channels; analyzing converted text through natural language processing (NLP) pipelines implementing intent recognition and sentiment analysis detecting emotional cues using a multimodal large language model (LLM); generating response content using machine learning models trained on domain-specific corpora; converting generated responses to synthetic speech through text-to-speech (TTS) engines; integrating with a customer relationship management (CRM) platforms or an enterprise resource planning (ERP) database; and implementing continuous learning by updating language understanding models using conversation logs, voice recognition parameters based on user feedback, and response generation patterns. One implementation is a computer-implemented system and method that operates a suite of intelligent interactive devices and platforms including an artificial intelligence voice agent, enhanced communication platforms, an intimacy companion system, and augmented / virtual reality eyeglasses. Further, one implementation includes AR / VR eyeglasses that project visual content onto interchangeable lenses or directly onto the user's retina via laser-based retinal projection, provide prescription adjustments, incorporate ear-mounted sensors for monitoring physiological parameters like heart rate, oxygen saturation, and blood pressure, and utilize wireless data transmission, onboard environmental sensing, and remote calibration, all designed to offer dynamically adaptive, secure, and context-aware interactions across communication, personal assistance, health monitoring, and immersive augmented or virtual reality environments.

Systems and methods are provided for machine learning models configured as zero-shot personalized text-to-speech models which comprise a feature extractor, a speaker encoder, and a text-to-speech module. The feature extractor is configured to extract acoustic features and prosodic features from new target reference speech associated with the new target speaker. The speaker encoder is configured to generate a speaker embedding corresponding to the new target speaker based on the acoustic features extracted from the new target reference speech. The text-to-speech module is configured to generate the personalized voice corresponding for the new target speaker based on the speaker embedding and the prosodic features extracted from the new target reference speech without applying the text-to-speech module on new labeled training data associated with the new target speaker.

The invention discloses a voice interaction method and device based on lip language enhancement, equipment and a storage medium, and the method comprises the steps: extracting lip language features based on an image sequence of a lip region, and carrying out the feature extraction of a voice signal, and obtaining an audio feature; performing cross-modal fusion coding on the lip language features and the audio features to generate mixed features containing audio-visual information; inputting the mixed features into a large language model, understanding the intention of the interaction object and generating a corresponding semantic reply; and finally, synthesizing into voice and / or converting into characters. According to the invention, by introducing the lip features, additional visual clues are provided for speech recognition, and the robustness and accuracy of speech recognition can be significantly improved; effective fusion coding is carried out on the lip language features and the sound features, and semantic information splitting caused by simple and independent recognition is avoided; and the capability of the large model is fully utilized, so that more natural and more intelligent interaction experience is realized.