7738 results about "Subvocal recognition" patented technology

The invention discloses a domain speech recognition method and system based on RAG. The method comprises the steps that an original domain document set is acquired, a hierarchical document structure is constructed through subject classification, paragraph segmentation and sentence segmentation, vectorization processing is executed to generate a multi-granularity vector matrix set, and a hierarchical knowledge vector index structure is established; performing multi-scale time-frequency analysis on the real-time audio stream, extracting and enhancing acoustic features, and generating a multi-scale feature tensor; performing adaptive blocking and progressive attention processing on the feature tensor, constructing a probability estimation model, and generating a candidate text sequence; executing multi-layer retrieval in the knowledge index based on the multi-dimensional query vector to generate related knowledge fragments; and carrying out space mapping and dynamic attention processing on the features, and executing probability optimization decoding to obtain a final recognition text. According to the invention, through knowledge enhancement and multi-level feature processing, the accuracy and robustness of voice recognition in the professional field are improved.

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

The invention relates to the technical field of intelligent conference processing, and relates to a conference summary processing method and system using AI, and the method comprises the steps: carrying out the real-time noise suppression of a collected conference audio stream and associated text data through a noise suppression algorithm, and carrying out the cross-modal alignment of the denoised data through a cross-modal alignment algorithm; a domain-specific attention head is inserted into an attention layer of the pre-trained Transform model, a domain-enhanced speech recognition model is constructed, and audio is converted into a text sequence with a speaker tag; adopting a heterogeneous graph neural network to construct a structured topic evolution graph; key decision nodes in the structured topic evolution graph are extracted based on a reinforcement learning strategy, and a final conference summary document is generated. In the decoding stage, the fusion proportion of the acoustic model and the language model is dynamically adjusted based on the real-time acoustic confidence coefficient, the recognition rate of the vocabularies in the professional field is increased, and the problems of frequent term transcription errors and poor semantic coherence in the professional conference are effectively solved.

A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments.

The invention provides an intelligent mechanical arm control system scheme based on multi-model cooperation, and belongs to the field of automation and intelligent control. The system is divided into four parts of man-machine interaction, detection and perception, task decoupling planning and program generation, and robot execution operation. Man-machine interaction is carried out, a user inputs an instruction in a text or voice format, and instruction processing is carried out through voice recognition and a natural language translation model; detecting and sensing, and identifying and positioning a target captured by the camera by using a multi-modal large model; task decoupling planning and program generation: analyzing the user instruction and the perception information into an executable program by applying a task planning model and a program translation model; and checking and correcting the program by using the program checking model, and driving the mechanical arm to complete a task. According to the system scheme, variable environments can be dealt with, complex language instructions can be understood, intelligent control over the mechanical arm is achieved, and the multi-task execution capacity and the intelligent level of the mechanical arm are remarkably improved.

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice intention recognition method, device and equipment and a medium, and the method comprises the steps: obtaining a to-be-processed voice signal, carrying out the voice activity detection processing of the voice signal, dividing the voice signal into a plurality of voice segments, analyzing semantic contents of the plurality of voice segments, determining semantic correlation information of each voice segment, analyzing sound source attributes of the plurality of voice segments, determining sound field type information of each voice segment, screening out a target voice segment from the plurality of voice segments according to the semantic correlation information and the sound field type information, and executing intention recognition processing based on the target voice segment to generate an intention recognition result. According to the invention, through a dual analysis mechanism of semantic correlation information and sound field type information, effective screening of voice segments is realized before voice recognition, non-target voice or interference segments are effectively prevented from being sent to an intention recognition model, and the accuracy of a recognition result is improved.

The invention provides a multi-modal intelligent question-answering and recommendation system supporting emotional speech output, and relates to the technical field of visual question-answering, and the system specifically comprises a multi-modal question-answering module, a multi-modal recommendation module, a speech recognition module, and a speech synthesis module. Wherein the multi-mode question answering module converts an image input by a user into a natural language form, and final question answering is completed in combination with a comet knowledge base COMET; the voice recognition module is used for recognizing voice data input into the multi-mode intelligent question answering and recommending system and processing the voice data into a text form; the multi-modal recommendation module identifies whether modal missing exists in input data or not, generates representation of the missing modal, obtains vector representation of the user and the article through a graph convolutional neural network and learning according to an interaction matrix of the user and the article, carries out reliability calculation, and finally sends a multi-modal recommendation result to the user and the article. Sending back to the multi-mode question and answer module; and the voice synthesis module outputs the voice to the user.

The invention discloses a large-model real-time voice text intention recognition method and system based on text prediction, and the method comprises the steps: obtaining the real-time voice data of a user, carrying out the real-time voice recognition processing through a streaming voice recognition interface, and obtaining a part of transcriptional text; inputting the partial transcription text into a mask language model for text prediction, and generating a plurality of high-credibility complete sentence candidates; based on the complete sentence candidates, the complete sentence candidates are input into a large language model in parallel for intention recognition, a corresponding intention result is obtained, and a mapping relation between the candidate sentences and the intention recognition result is established; and obtaining a sentence completely expressed by the user, calculating the similarity between the complete actual sentence and a plurality of high-credibility complete sentence candidates through a multi-level text similarity algorithm, selecting the candidate sentence with the highest similarity score, and directly obtaining a corresponding final intention recognition result based on the mapping relationship. The objective of the invention is to solve the technical problem of high response delay of an existing voice intention recognition system.

The invention relates to an intelligent equipment access system and access method based on an AI large model, and relates to the field of artificial intelligence and Internet of Things. According to the technical scheme, a dynamically adaptive universal object model access framework is constructed, functional attributes, control instructions and data formats of intelligent devices of different brands and protocols are mapped into a standardized logic model, meanwhile, a multi-mode AI large model engine is combined, high-precision voice recognition, voiceprint feature detection and multi-level semantic intention analysis capabilities are integrated, and the intelligent devices of different brands and protocols can be obtained. And an intelligent interaction closed loop of end-cloud collaboration is formed. Semantic analysis adopts a mixed strategy, a natural language instruction is converted through function mapping, a local knowledge base is utilized to match user scene requirements, and external data is queried by means of cloud networking to optimize instruction logic. According to the scheme, the multi-device access and operation and maintenance cost is remarkably reduced, the semantic understanding accuracy and scene adaptability are improved, and the method is widely applied to the fields of smart home, smart medical treatment and industrial Internet of Things.

The invention discloses a speech recognition and natural language processing integration method and system, and the method comprises the steps: carrying out the multi-modal data fusion according to a speech signal of a user, context text information and environment sensor data, and obtaining a fused multi-modal feature vector; inputting the multi-modal feature vector into a speech recognition model based on an adaptive deep neural network, and performing speech-to-text processing to obtain text output; inputting the text output into a semantic analysis model based on a graph neural network, and performing context semantic analysis and user intention recognition to obtain semantic representation of the user intention and a confidence score of the semantic representation; and according to the semantic representation and the confidence score thereof, dynamically adjusting parameters of the speech recognition model and the semantic analysis model by using a feedback optimization technology based on reinforcement learning, and generating a model optimization strategy. According to the embodiment of the invention, the accuracy of speech recognition and the semantic comprehension capability of natural language processing can be improved.

This application discloses a voiceprint recognition method, a graphical interface, and an electronic device. In the voiceprint recognition method, a voiceprint model is preset in the electronic device, and then the electronic device trains and updates the preset voiceprint model based on a voiceprint feature extracted from a voice of a registered user to obtain an exclusive voiceprint model belonging to the registered user. Finally, the electronic device uses the exclusive voiceprint model to generate a registered user representation based on the voiceprint feature of the voice of the registered user, and uses the registered user representation as a reference standard to realize voiceprint recognition on a voice of a speaker. Since the exclusive voiceprint model is trained based on voiceprint features of personal voices of the registered user, the registered user representation generated can accurately express voiceprint features of the user, thereby improving accuracy of voiceprint recognition.

The invention relates to the technical field of medical informatization, and discloses a method for generating a medical record report based on a doctor-patient dialogue, and the method comprises the following steps: multi-modal data collection: collecting the multi-modal data of the doctor-patient dialogue, including voice data, video data and environment data; multi-modal data preprocessing: preprocessing the collected data, including noise reduction and feature extraction of voice signals, key point extraction of video data and time synchronization of the multi-modal data; and semantic analysis: converting the voice signal into a text by using a voice recognition technology, carrying out semantic analysis on the text, and extracting key information. Through multi-modal data acquisition, semantic analysis, emotion modeling and knowledge graph reasoning technologies, the comprehensiveness, logicality and dynamic correction ability of medical record generation are improved, meanwhile, the defects of patient psychological state analysis in the prior art are overcome, information orderliness and doctor decision-making efficiency are remarkably improved through generation of the visual medical record report, and the medical record report generation method is suitable for popularization and application. And powerful support is provided for precise medical treatment.

The invention belongs to the technical field of speech recognition. The invention provides a low-resource language self-adaptive speech recognition method based on AdLoRA Plus. The low-resource language self-adaptive speech recognition method comprises the following steps of: obtaining a speech According to the embodiment of the invention, the AdLoRA Plus fine tuning technology is introduced, so that the speech recognition precision of the Whisper model is remarkably improved in a low-resource language environment. Through a low-rank adapter layer and dynamic learning rate adjustment, efficient parameter updating is realized under limited training data. According to the method, a strategy of freezing basic parameters is adopted, only an AdLoRA Plus adapter layer is trained, calculation overhead and training time are remarkably reduced, pre-training knowledge is kept, and overfitting is prevented. The method has wide adaptability and expansibility, and is ensured to be outstanding in practical application. The voice recognition precision, the reasoning efficiency and the training efficiency are effectively improved, multiple challenges in application in small languages and specific fields in the prior art are solved, and the method has important practical application value.

The invention discloses an intelligent auxiliary teacher lesson preparation system and method based on an education large model. The method comprises the following steps: S1, generating a standardized teacher lesson preparation data set; s2, constructing a teaching knowledge graph; s3, generating preliminary intelligent lesson preparation content; s4, presenting the preliminarily generated intelligent lesson preparation content, and providing an interactive interface combining a natural language and voice recognition; s5, generating the adjusted intelligent lesson preparation content; and S6, according to historical lesson preparation data of the teacher and the interactive feedback data, performing adaptive learning and optimization on the intelligent lesson preparation content, adjusting weights of key knowledge points in the teaching knowledge graph, updating a content generation strategy of the large education model, and generating a final intelligent lesson preparation scheme. According to the method, the response capability of the intelligently generated content to the teaching intention of a teacher is remarkably improved, the teaching content and the test question style are supported to dynamically cooperate with classroom feedback along with the teaching rhythm, and an intelligent decision support basis is provided for large-scale personalized teaching.

A computer-implemented method for operating an artificial intelligence voice agent system includes receiving voice input through communication channels; analyzing converted text through natural language processing (NLP) pipelines implementing intent recognition and sentiment analysis detecting emotional cues using a multimodal large language model (LLM); generating response content using machine learning models trained on domain-specific corpora; converting generated responses to synthetic speech through text-to-speech (TTS) engines; integrating with a customer relationship management (CRM) platforms or an enterprise resource planning (ERP) database; and implementing continuous learning by updating language understanding models using conversation logs, voice recognition parameters based on user feedback, and response generation patterns. One implementation is a computer-implemented system and method that operates a suite of intelligent interactive devices and platforms including an artificial intelligence voice agent, enhanced communication platforms, an intimacy companion system, and augmented / virtual reality eyeglasses. Further, one implementation includes AR / VR eyeglasses that project visual content onto interchangeable lenses or directly onto the user's retina via laser-based retinal projection, provide prescription adjustments, incorporate ear-mounted sensors for monitoring physiological parameters like heart rate, oxygen saturation, and blood pressure, and utilize wireless data transmission, onboard environmental sensing, and remote calibration, all designed to offer dynamically adaptive, secure, and context-aware interactions across communication, personal assistance, health monitoring, and immersive augmented or virtual reality environments.

The invention discloses a voice interaction method and device based on lip language enhancement, equipment and a storage medium, and the method comprises the steps: extracting lip language features based on an image sequence of a lip region, and carrying out the feature extraction of a voice signal, and obtaining an audio feature; performing cross-modal fusion coding on the lip language features and the audio features to generate mixed features containing audio-visual information; inputting the mixed features into a large language model, understanding the intention of the interaction object and generating a corresponding semantic reply; and finally, synthesizing into voice and / or converting into characters. According to the invention, by introducing the lip features, additional visual clues are provided for speech recognition, and the robustness and accuracy of speech recognition can be significantly improved; effective fusion coding is carried out on the lip language features and the sound features, and semantic information splitting caused by simple and independent recognition is avoided; and the capability of the large model is fully utilized, so that more natural and more intelligent interaction experience is realized.

The invention relates to the technical field of audio processing, and discloses an audio processing method and system for speech spectrum reconstruction, and the method comprises the steps: carrying out the preprocessing of an original speech signal collected by a vehicle-mounted microphone, and obtaining a speech signal time-frequency matrix; performing singular value decomposition on the voice signal time-frequency matrix to obtain a sparse matrix representing voice components and a low-rank matrix representing noise components, and establishing a frequency band characteristic incidence matrix through a mutual information entropy calculation method; inputting the sparse matrix into a selection forgetting extreme learning machine for voice feature prediction to obtain predicted voice feature data; and based on the predicted voice feature data and the frequency band feature incidence matrix, performing voice probability calculation and feature interpolation on each time frequency point by adopting a logarithmic frequency spectrum amplitude calculation method to obtain reconstructed frequency spectrum data, thereby improving the accuracy of voice recognition and meeting the requirement of real-time processing in a vehicle-mounted environment at the same time.

Automatic recognition of non-verbal messages in speech, and in particular to detection or analysis of prosodic multilayered analysis of intonation units such as prosodic unit prototypes and their multi-labeled variations, may form a hierarchical classification for the analysis of non¬ verbal information or cues in speech. A speech captured by a microphone is fed to a weakly- supervised deep learning acoustic model for speech recognition and transcription, that may be based on encoder-decoder Transformer architecture, such as Whisper by OpenAI. The model is trained to output multiple words form the text in the captured speech, to identify Intonation Units (IUs) that include one or more words, and associate non-verbal labels to each of the IUs. The labels may indicate a prototype, a discourse function (such as a conversation action), an emotion, an emphasis, or an attitude, as well as a genre of a part of, or whole of, the entire captured speech.

The invention relates to the technical field of voice interaction, particularly discloses an AI intelligent robot voice interaction method and system, and aims to solve the problems of low voice interaction accuracy, insufficient reliability and lack of authority control in a complex noise environment. A dynamic noise feature library containing steady-state noise, impact noise and human voice interference features and a pre-stored gesture instruction library are constructed, audio signals are collected in real time, low-frequency-band, middle-frequency-band and high-frequency-band differential noise reduction is executed, Mel-frequency cepstral coefficient features are extracted, noise scenes are matched, corresponding voice recognition models are switched, and voice recognition is achieved. And calculating a confidence value of the voice instruction, outputting multi-modal verification data in combination with a dynamic confidence threshold, and outputting an authority control signal through voiceprint matching, authority verification and instruction consistency judgment. Through multi-modal fusion, dynamic adaptation and authority control, the voice recognition accuracy and interaction safety in a complex noise environment are remarkably improved, and the method is suitable for scenes such as factory intelligent inspection.

The invention discloses a voice interaction optimization method and system based on a multi-modal large model, and relates to the technical field of artificial intelligence and voice interaction, and the method comprises the steps: carrying out the voice recognition in response to the real-time voice of a user, and obtaining voice text information; according to the voice text information, combining the voice waveform of the real-time voice of the user as the input of a multi-modal recognition model, so as to judge whether the voice dialogue is interrupted and recognize the interruption intention, and obtaining a voice interruption result; obtaining a new intention of the user according to a voice interruption result, and dynamically adjusting a system response strategy to realize interaction optimization; the accuracy and comprehensiveness of interruption detection are improved through multi-modal fusion, so that the interruption intention is accurately recognized to respond to the user intention in real time, the system dialogue interaction efficiency and reliability are remarkably improved, and the defects that existing voice interruption detection is not high in detection accuracy, dynamic response of user interruption behaviors cannot be achieved, and user experience is poor are overcome. Therefore, the problems of low interaction efficiency and poor reliability of the voice dialogue system are solved.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech recognition error correction method, which comprises the following steps of: training and generating a target field Chinese character text understanding model by collecting dialogue Chinese character texts of a target field; collecting pinyin data, and training to generate a pinyin understanding model; receiving to-be-detected voice data and converting the to-be-detected voice data into to-be-detected Chinese character texts and pinyin texts Inputting the Chinese character text and the pinyin text to corresponding understanding models to generate semantic features and pinyin features; combining the semantic features and the pinyin features to form joint features; inputting the joint features into a speech recognition error detection model, generating error probability distribution of each word, and outputting an error tag; and correcting the Chinese character text through a template error correction module based on the error tag. According to the method, more comprehensive feature representation is formed by combining semantic features and pinyin features, and the accuracy of speech recognition error detection is improved; and a template error correction module is utilized to effectively correct identification errors.

The invention discloses a speech recognition and authentication method and system based on multi-modal features and dynamic evaluation in the technical field of speech recognition and authentication, and the method comprises the steps: collecting an original speech signal of a user through a microphone, and carrying out the preprocessing of the original speech signal, and obtaining the preprocessing speech data; and extracting feature data of the preprocessed voice data by adopting a multi-dimensional feature hierarchical extraction technology, and injecting a multi-source noise sample into an acoustic feature space of a voiceprint feature model based on an initial training stage established by the voiceprint feature model to construct an anti-noise mixed voiceprint map. Through integrating voiceprint, semantics, behavior characteristics and an environment adaptation mechanism, an authentication threshold is adjusted in real time according to dynamic risk assessment, meanwhile, a risk scoring model is utilized to calculate a comprehensive risk value, authentication modes of different levels are started according to risk scenes of different degrees, and two-factor authentication is forcibly implemented for high-risk scenes. And the authentication security and reliability can be obviously enhanced.

Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.

The invention discloses a method for supporting fast switching of multiple models by NPU (Network Processing Unit) firmware. According to the invention, the system real-time performance and the resource utilization rate in a multi-task scene are obviously improved. Through a dynamic hierarchical caching strategy and a priority scheduling mechanism, the system can intelligently allocate cache resources and preferentially guarantee rapid loading of a high-frequency and high-urgency model, for example, in an automatic driving scene, switching delay of a path planning model can be reduced to a millisecond level, and non-perceptual switching of key tasks is guaranteed. The incremental parameter loading technology and firmware-level context management are deeply fused, only model difference data are transmitted, and hardware is utilized to accelerate and recover a calculation state, so that bandwidth waste caused by traditional full-amount loading is greatly reduced, the model switching efficiency of edge computing equipment is improved by more than 10 times when multiple tasks such as voice recognition and image processing are carried out in parallel, and the efficiency of the edge computing equipment is improved. And meanwhile, the calculation precision is kept lossless.

The invention discloses a construction method of a real-time streaming voice intelligent question and answer service system. The construction method comprises the following steps: receiving input voice data; calling an independent voice recognition service to convert the voice data into an input text; the method comprises the following steps: constructing cue words of a large language model based on an input text, initiating an LLM streaming request to an independent large language model service, and obtaining streaming text answers generated by LLM in real time; segmenting the flow type text answer in real time through a dynamic sentence segmentation device to generate a plurality of clauses; calling an independent speech synthesis service in parallel for each clause, and converting the text into an audio data block; and combining the audio data blocks into streaming audio data according to the generation sequence, and returning the streaming audio data to a client for playing in real time. According to the invention, by decoupling ASR, LLM and TTS services and combining the asynchronous streaming framework and the strategy of "wanting while speaking", the real-time performance and flexibility of the voice question-answering system are significantly improved.

The invention discloses a language learning assistance application system based on speech recognition, and belongs to the technical field of learning assistance. The system comprises an intelligent voice acquisition and enhancement module for realizing acquisition and preprocessing of high-quality voice signals in a complex environment; the multi-modal feature extraction module dynamically extracts and fuses multi-dimensional learning feature information, completes feature analysis and boundary segmentation of a phoneme level, and is responsible for feature normalization and optimization; the deep speech recognition module is used for realizing high-precision speech recognition based on the normalized features; the intelligent evaluation engine module is used for realizing real-time and multi-dimensional evaluation on the language performance of the learner; the personalized learning management module integrates the evaluation results for dynamically planning an optimal learning path for the learner, manages the learning progress, and provides learning effect prediction and intervention suggestions at the same time; and the learning interaction experience module is responsible for visual presentation of learning feedback, multi-modal man-machine interaction design and user interface optimization.

The invention discloses a voice recognition model optimization method and device based on reinforcement learning, equipment and a medium, and relates to the technical field of voice recognition, and the method comprises the steps: determining a current optimization strategy of a current time step; randomly selecting one piece of enhanced speech training data and corresponding real text training data from the data training set as target training data, and inputting the target training data into the current automatic speech recognition model for prediction to obtain a plurality of prediction texts output by the current automatic speech recognition model; determining a text matching degree and a semantic similarity between each prediction text and real text training data to obtain a group of target reward values, updating an optimization strategy at the current moment based on the target reward values, and skipping to execute the step of determining the current optimization strategy of the current time step to obtain a target reward value; and outputting the corresponding optimized automatic speech recognition model until the current automatic speech recognition model meets the optimization ending condition. And reasonable optimization of the automatic speech recognition model is realized.

The invention discloses a holographic display digital human speech recognition enhancement method based on multi-modal interactive learning, and the method comprises the following steps: S1, collecting and preprocessing data, and generating a multi-modal feature matrix; s2, inputting the multi-modal feature matrix into a self-adaptive multi-modal interaction kernel to generate multi-modal interaction features; s3, inputting the multi-modal interaction features into an emotion convolution block to generate semantic-emotion mapping features; s4, inputting the semantic-emotion mapping features into a cross attention generator, and outputting multi-modal fusion features; s5, inputting the multi-modal fusion features into a multi-modal fusion-decomposition two-way network, and generating multi-modal features after decoupling optimization; s6, inputting the multi-modal features into a holographic-speech symbiosis optimization module to generate speech recognition and holographic display data; and S7, performing parameter optimization by applying self-supervised learning and comparative learning strategies to realize self-adaptive adjustment. According to the invention, the enhancement of holographic display digital human speech recognition is realized by using a multi-mode interactive learning method.

The invention provides an online car-hailing passenger position rapid positioning method based on voice interaction and visual perspective, and relates to the technical field of intelligent traffic, and the method comprises the following steps: arranging a sensor, a camera, a metal reflective curtain and a projector on a car; voice information of a driver is collected, and an intelligent voice wake-up model is designed; constructing a scenarized speech recognition model; detecting a pedestrian target for the panoramic image by using a sound-image multi-level cross-modal detection model, and obtaining appearance and posture feature detection information of the pedestrian target; a static feature matching method and a dynamic feature recognition model are designed, and passenger feature information matching is achieved; based on a matching result, determining position information of the passenger; acquiring a passenger direction taking the vehicle as a reference origin through position estimation in the panoramic image; according to the direction estimation result, a projector in the vehicle is controlled, the passenger information is projected on a curtain in the corresponding direction in the vehicle, and meanwhile voice information is used for giving a prompt.

The invention provides a Zip-MoE model grouping hybrid expert layer-based Chinese and English speech recognition method and system, a Zip-MoE model comprises six encoder blocks, a Bypass module is included between every two encoder blocks, and the weights of the output of the previous encoder block and the output of the current encoder block are learned; the first three encoder blocks are of a standard Zipform structure; the last three encoder blocks adopt a Zipform-MoE structure containing a grouped hybrid expert layer, and the grouped hybrid expert layer is used for replacing the last feed-forward network of the Zipform structure; the grouping mixed expert layer comprises a Chinese expert group, an English expert group and a language router, and each expert group is composed of a plurality of expert networks and is provided with an unsupervised router. The problem of language confusion is relieved, different time delay streaming scenes can be adapted, the number of experts is flexibly expanded, pre-training is not needed, and the Chinese and English recognition efficiency is greatly improved.