66 results about "Modal voice" patented technology

The invention provides a multi-mode voice interaction method based on a large model, electronic equipment and a storage medium, and the method comprises the steps: carrying out the voice recognition of the voice data of a first language inputted by a user, and obtaining a first language text; performing language translation on the first language text to translate the first language text into a second language text; performing semantic analysis on the second language text to obtain a semantic analysis result; performing multi-modal knowledge graph construction on the semantic analysis result based on a preset large model to obtain an enhanced semantic understanding result; wherein the multi-modal information related to the second language text is fused in the multi-modal knowledge graph; generating a dialogue strategy of the enhanced semantic understanding result; and performing voice synthesis based on the dialogue strategy to generate reply voice data expressed in the first language. According to the method and the device, multi-language interaction is realized, and meanwhile, various modal information is fused.

The invention relates to the technical field of digital interaction, in particular to a digital interaction enhancement system based on multi-modal voice, which comprises a multi-modal sensing module comprising a voice sensing unit, a visual sensing unit and an environment sensing unit and used for capturing multi-modal data input by a user; the multi-modal fusion module comprises a data synchronization unit, a feature extraction unit and a multi-modal fusion unit and is used for performing time synchronization, feature extraction and fusion processing on the multi-modal data; and the interactive understanding module comprises a natural language processing unit, a context understanding unit and an emotion analysis unit and is used for understanding the intention and the emotion state of the user. The deep learning technology is adopted to fuse voice, vision and environment data, and efficient and natural user interaction experience is achieved through the steps of feature extraction, feature fusion, intention recognition and the like. The system has excellent performance and a wide application prospect in a complex environment.

A lightweight multi-modal speech enhancement system for head-mounted wearables, called VibVoice. A two-branch encoder-decoder deep neural network is employed to exploit the complementary characteristics of two modalities to reconstruct clean speech: (i) acceleration, which has a low sample rate covering a partial audible spectrum but with no environmental noises, and (ii) microphone, which has a high sample rate covering the whole audible spectrum but is mixed with ambient voice and noises. To challenges the issue of insufficient paired data for training, embodiments extensively measure the bone conduction effect from a limited dataset to extract a physical impulse function for cross-modal data augmentation.

The invention discloses a privacy information desensitization method and system for a voice generation type large model, and relates to the technical field of artificial intelligence. Input voice data is discretized, Gaussian noise disturbance sensitive features are injected, and a cross-modal voice generation model is constructed in combination with three-stage training; and meanwhile, a cross-modal privacy enhancement mechanism is applied to detect and fuzzify sensitive information in real time in an output stage. According to the invention, the adaptability of a large-scale voice generation model in a privacy protection scene is improved, and comprehensive protection of user privacy is realized. And moreover, the capability of extracting user privacy information by an adversarial attacker is effectively limited, and the risk of sensitive information leakage in the data transmission, storage and generation process of the voice generation model is reduced. On the premise that privacy is ensured, the voice generation model can still keep high-quality generation performance, generated voice output has high naturalness and accuracy, and actual application requirements are met.

The invention relates to the technical field of speech recognition, and particularly provides a multi-mode speech processing method and device, a medium and a computer program product. The multi-mode voice processing method comprises the following steps: acquiring to-be-processed audio and video data, and performing data separation processing on the to-be-processed audio and video data to obtain original audio data and original video data; performing face detection processing on the original video data to obtain initial video data; performing audio detection processing on the original audio data to obtain initial audio data; and performing consistency detection processing on the initial video data and the initial audio data, and outputting target audio data and target video data under the condition that a consistency verification condition is met. The face in the video is recognized through face detection, noise and sound of the target speaker are recognized and separated through audio detection, and synchronization and matching of audio and video data in time and content are ensured through consistency detection, so that development of a multi-mode voice separation task is supported.

The invention provides an AI multi-mode voice interaction method based on a vehicle-mounted intelligent terminal and electronic equipment, and the method comprises the steps: synchronously collecting voice information, facial micro-expressions and hand touch actions of a user through a microphone array, an in-vehicle camera and a steering wheel touch sensor on the vehicle-mounted intelligent terminal, and forming multi-mode interaction data; fusing the multi-modal interaction data, and judging a current driving scene through a pre-trained driving scene recognition model; dynamically adjusting voice interaction parameters according to the driving scene judgment result; recognizing a voice instruction of the user based on the adjusted voice interaction parameter; and in combination with instruction preferences in historical interaction data of the user, intention completion is performed on the recognized voice instruction, and a personalized execution instruction is generated. According to the invention, the recognition accuracy and interaction efficiency of the voice instruction are improved.

The invention discloses a multi-mode voice interaction system and method for an intelligent cabin. Firstly, various vehicle working condition parameters in a CAN bus are read in real time, a wavelet domain Wiener filtering and judgment statistic enhancement strategy is dynamically adjusted, and self-adaptive voice purification based on vehicle state prior is achieved. Secondly, a bidirectional long-short-term memory network is adopted as a semantic understanding core, semantic feature vectors are extracted from enhanced signals through segmented maximum pooling operation, and the problem of semantic confusion caused by manifold collapse in a traditional method is effectively solved. And the feature vector and a safety state signal are synchronously input into an improved LLM-CoT engine and a dynamic arbiter, so that context accurate understanding and safety-up priority response are realized. According to the method, vehicle working condition perception and biLSTM depth time sequence modeling are deeply adapted to a vehicle-mounted complex acoustic scene, the error rate of semantic understanding is remarkably reduced in actual measurement under multi-person dialect dialogue, and the accuracy, robustness and safety of intelligent cabin interaction are systematically improved.

The invention provides a voice separation method and system, a vehicle and a storage medium. The method comprises the steps of collecting an image sequence and a mixed voice signal of at least one object in a vehicle space; and performing voice signal separation according to the image sequence and the mixed voice signal to determine a voice signal of each object in the vehicle control. According to the scheme, a traditional scheme of configuring a plurality of voice acquisition devices for voice acquisition to ensure accurate voice separation is abandoned, and multi-mode voice separation is performed by acquiring the corresponding image sequence and combining the mixed voice signals, so that the voice separation accuracy is improved, the dependence of voice separation on voice acquisition equipment is reduced, and the voice separation efficiency is improved. Configuration of the voice acquisition device can be reduced, and the cost is reduced.

The invention relates to the technical field of voice signal processing, and discloses a method for realizing personalized singing synthesis model training through multi-modal voice driving, and the method comprises the following steps: obtaining multi-modal input data, including text, reference audio, speaker and emotional features; processing the data to obtain coding features; the inter-modal redundant information is evaluated and suppressed through redundant perception coding; compressing coding features by using an information bottleneck model, and retaining effective information; fusing the compressed features to generate personalized singing features; the input decoding module generates Mel spectrum features; and converting the Mel spectrum into an audio waveform through a vocoder, and outputting the audio waveform. And converting the Mel spectrum into an audio waveform through a vocoder, and outputting the audio waveform. According to the method, multi-modal data can be effectively fused, the individuation and emotion expression ability of the singing sound is improved, redundant information interference is reduced, the quality and efficiency of singing sound synthesis are improved, and finally, high-fidelity individualized singing sound is output.

The invention belongs to the technical field of speech analysis, and discloses a spoken English pronunciation quality evaluation method based on multi-modal speech feature analysis, which comprises the following steps: acquiring a spoken English speech signal of a target user, and performing multi-domain decomposition on the speech signal to obtain multi-modal speech features; performing time-frequency domain corresponding relation analysis and feature extraction on the multi-mode speech features to obtain a pronunciation detail feature sequence; performing multi-scale matching on the pronunciation detail feature sequence and a preset standard pronunciation template, constructing a multi-dimensional representation model based on a multi-scale matching result, and calculating a fine-grained quality score of a phoneme unit in each multi-dimensional representation model in combination with rhythm and rhythm parameters in the multi-modal speech features, the rhythm coherence score and the overall fluency score are fused to generate a comprehensive pronunciation quality evaluation result and a visual diagnosis report of pronunciation deviation; the oral English pronunciation evaluation method realizes comprehensive and refined evaluation of oral English pronunciation, and provides a scientific guidance basis for personalized language learning.

The invention relates to the technical field of artificial intelligence and multimedia, and particularly discloses a video generation method based on voice analysis, and the method comprises the following steps: analyzing an input voice, and extracting a multi-modal voice feature; inputting the multi-modal speech features into a pre-trained scene association model, and outputting a scene label set; on the basis of dialect features in the input voice, region categories are recognized through a dialect classifier, and a corresponding visual element library is loaded from a culture database according to the region categories; selecting a scene template according to scene type labels in the scene label set, and selecting a character action template in combination with emotion type labels and interaction object relation labels; and calculating time sequence distribution of the video elements based on the speech speed change parameters, and rendering the scene template, the character action template and the rhythm of the input speech according to the time sequence distribution through a time sequence alignment algorithm to generate a target video. The method can improve the accuracy of the video generated based on the voice.

The invention discloses a multi-modal voice interaction large model training method and system based on voice acoustic feature regulation and control, terminal equipment and a medium, and relates to the technical field of multi-modal voice interaction.The method comprises the steps that a text token of a text training sample is obtained, a corresponding voice token is constructed, and pre-training data used for converting the text token into the voice token is obtained; in combination with the multi-modal input sample and the pre-training data, constructing fine-tuning training data for speech understanding and dialogue generation; constructing and pre-training a basic model by using the pre-training data; a multi-modal voice interaction large model is constructed based on a pre-training basic model, and fine tuning data is used for training, so that voice acoustic features can be regulated and controlled based on multi-modal input, and voice is output. According to the method, through alignment and staged training of the text token and the voice token, fine regulation and control of voice acoustic characteristics are realized, long voice continuity and interaction naturalness are improved, and a model is efficiently endowed with voice interaction capability of controllable timbre and emotion.

The invention provides a multi-modal speech enhancement algorithm based on semi-supervised learning pre-training. The multi-modal speech enhancement algorithm comprises the following steps: S1, taking an extracted noisy speech feature vector # imgabs0 # as the input of a noisy speech GCRBM; s2, de-noising the noisy voice # imgabs 1 # by using a traditional spectral subtraction method to obtain a de-noised voice # imgabs 2 #; s3, taking the estimated de-noised speech feature vector # imgabs3 # as the input of the pure speech GCRBM; s4, taking the extracted video signal feature vector # imgabs4 # as the input of the video GCRBM; s5, optimizing parameters of the multi-mode speech enhancement network; s6, during testing, noise-containing speech features are input to the multi-mode speech enhancement network; according to the method, the voice features and the mouth features are extracted and input into the multi-mode voice enhancement network, and after parameter optimization of the multi-mode voice enhancement network is carried out, the noisy voice features are input into the multi-mode voice enhancement network to obtain the enhanced voice signals, so that the voice annotation cost is reduced, and the intelligibility of voice enhancement is improved.

The invention discloses a method and a device for predicting voice content based on a multi-modal voice recognition model, and the multi-modal voice recognition model comprises a voice-to-text large model, a modal adaptation layer, an Embedding layer and a large language model. The training process of the multi-modal speech recognition model comprises the following steps: a first training stage: training the multi-modal speech recognition model by using automatic speech recognition data, keeping model parameters of a large language model and an Embedding layer fixed, and training a speech-to-text large model and a modal adaptation layer; and a second training stage: after the first training stage is completed, keeping model parameters of the voice-to-text large model obtained by training in the first training stage fixed, adopting an optimization fine tuning strategy for the large language model, and training a multi-modal voice recognition model by using automatic voice recognition data which is the same as that in the first stage. According to the method for predicting the voice content based on the multi-modal voice recognition model, the technical problem that parameters of a fixed language model cannot well adapt to actual scene requirements is solved.

The invention provides a multi-mode voice interaction method and device and intelligent equipment, is suitable for the technical field of intelligent voice interaction, is applied to the intelligent equipment, and comprises the following steps: in response to a detected voice activity, extracting first voice data in the voice activity, and obtaining video data shot synchronously with the first voice data, a plurality of different users are shot in the video data. And screening out a target user sending the first voice data from the video data. And performing semantic integrity analysis on the text content corresponding to the first voice data. And when the semantic integrity analysis result is that the semantics is incomplete, continuously acquiring second voice data of the target user for a plurality of times, and generating a corresponding target statement with complete semantics after the first voice data and the second voice data are combined. Generating reply data according to the target statement, and outputting the reply data through voice. According to the embodiment of the invention, accurate, coherent and real-time voice interaction with the user needing interaction can be realized.

The invention relates to the technical field of voice recognition, and particularly provides a multi-modal voice recognition method based on a large model, a storage medium, electronic equipment and a product, and the method can comprise the steps: carrying out the preprocessing of an original voice signal of a user, and obtaining a processed voice signal; inputting the voice coding data and the historical dialogue data corresponding to the processed voice signal into a large language model to obtain a text vector corresponding to the processed voice signal; performing feature extraction on the processed voice signal to obtain a voice feature vector; processing a target vector sequence formed by splicing the voice feature vector and the text vector by using a pre-trained voice recognition module to obtain a text sequence; wherein the voice recognition module comprises a plurality of encoder layers and a plurality of decoder layers which are trained in advance; and cleaning and formatting the text sequence to obtain text data corresponding to the original voice signal. Some embodiments of the present application can improve the accuracy of speech recognition.

The invention proposes a voice interaction recognition enhancement method, and the method comprises the steps: collecting a video of voice interaction of a speaker facing a camera, dividing the video into N segments of to-be-recognized voices and N frames of to-be-recognized images, and forming to-be-recognized data; inputting the to-be-recognized data into a preset voice interaction recognition enhancement model which comprises a lip language feature extraction network, a voice feature extraction network, a time feature extraction network and an activation network, the time feature extraction network is used for adding time sequence information to a lip language feature matrix extracted by the lip language feature extraction network and a voice feature matrix extracted by the voice feature extraction network, and the activation network is used for simulating an activation-suppression mechanism of visual information in a biological brain on an auditory neural circuit, so that interaction between a visual mode and an auditory mode is realized. And thus, a speech recognition result is obtained. According to the invention, visual and auditory dual-mode speech recognition is realized, lip language and speech are accurately recognized in a noisy environment, and the response capability and accuracy meet requirements.

A multi-modality voice recognition device is provided. The multi-modality voice recognition device includes a video encoder trained to receive a lip video to a video encoder to extract visual feature information for voice recognition, an audio encoder trained to receive a voice to extract voice feature information for voice recognition, a modality reconstructor trained to reconstruct the voice feature information from the visual feature information to generate reconstruction voice feature information, a random selector configured to randomly output one of the voice feature information and the reconstruction voice feature information, and a video-audio decoder trained to receive a multi-modality feature, where the visual feature information is connected to an output of the random selector, to output a character string which is a voice recognition result.

The invention relates to multi-mode voice triggering for audio devices. Particular implementations of the subject technology provide systems and methods for multi-mode voice triggering of audio devices. An audio device may store a plurality of speech recognition models, each of which is trained to detect a single corresponding trigger phrase. In order that an audio device can detect a particular one of a plurality of trigger phrases without consuming processing and / or power resources to run a speech recognition model that can distinguish different trigger phrases, the audio device preloads a selected one of the speech recognition models for an expected trigger phrase into a processor of the audio device. The audio device may select one of the speech recognition models for the expected trigger phrase based on a type of companion device communicatively coupled to the audio device.

The invention discloses a shopping guide capability improving system and method based on an AI voice work card, and belongs to the technical field of artificial intelligence and intelligent shopping guide. The system comprises a multi-mode voice acquisition and feature extraction module, a crown cancellation model dynamic extraction and scoring module, a dialogue quality evaluation and intention analysis module and a closed-loop feedback optimization and partner training scene generation module, and the four modules form a deep coupling closed-loop cooperative system. The system extracts efficient verbal skills from crown sales samples through an innovative adaptive weight calculation algorithm, quantifies customer intention intensity through a multi-dimensional fusion method, improves shopping guide ability through interactive partner training, realizes system adaptive optimization through a closed-loop feedback mechanism, and improves customer experience. According to the invention, automatic inheritance of sales crowd experience and scientific evaluation and systematic improvement of shopping guide capability are realized, customer satisfaction and sales conversion rate are significantly improved, and the method is suitable for industries requiring shopping guide service, such as extensive home and retail.

The present application belongs to the technical field of speech analysis, and discloses an English oral pronunciation quality evaluation method based on multi-modal speech feature analysis, which comprises the following steps: obtaining the speech signal of the target user's English oral speech, and performing multi-domain decomposition on the speech signal to obtain multi-modal speech features; performing time-frequency domain corresponding relationship analysis and feature extraction on the multi-modal speech features to obtain a pronunciation detail feature sequence; performing multi-scale matching on the pronunciation detail feature sequence and a preset standard pronunciation template, and constructing a multi-dimensional representation model based on the multi-scale matching result; calculating the fine-grained quality score of the phoneme unit in each multi-dimensional representation model in combination with the prosody rhythm parameters in the multi-modal speech features; and fusing the prosody coherence score and the overall fluency score to generate a comprehensive pronunciation quality evaluation result and a visual diagnostic report of pronunciation deviation; the present application realizes comprehensive and refined evaluation of English oral pronunciation, and provides scientific guidance basis for personalized language learning.

The invention discloses a method and system for improving the voice inter-translation quality of a small language based on a large language model, and belongs to the technical field of intelligent voice translation services. The method comprises the following steps of language information input, speech detection and acquisition, semantic feature and acoustic feature extraction, automatic speech recognition (ASR) model recognition, large language model (LLM) translation and error correction, multi-modal speech synthesis (TTS) model fusion and target language speech signal output. The system comprises the following modules: a language information input module, a voice detection and acquisition module, a voice coding module, an automatic voice recognition module, a large language model translation and error correction module, a multi-modal voice synthesis module and a voice output module. According to the method, the accuracy and the naturalness of the voice inter-translation of the minority language can be remarkably improved, the method adapts to the characteristics of the multi-language minority language of the East Alliance, the industrial-grade application requirements are met, and the problems of low recognition accuracy and acoustic feature loss of the voice inter-translation of the minority language are solved.

The application discloses a multi-modal speech separation method, a training method and related devices. The method comprises: obtaining audio and video data containing a target object; wherein the audio and video data contains lip video data of the target object; inputting the audio and video data into a trained multi-modal speech separation network to obtain audio data related to the lip video data of the target object; wherein a plurality of training samples for training the multi-modal speech separation network are divided into a plurality of subsets based on a first loss obtained after the training samples pass through the multi-modal speech separation network, and the multi-modal speech separation network is trained again based on at least part of the subsets. In the foregoing manner, the application can improve the accuracy of multi-modal speech separation.

The invention relates to the technical field of face recognition, and discloses a multi-modal voice interaction method based on face recognition, which comprises the following steps of: 1, acquiring face features, and recording and numbering a plurality of face features; step 2, face analysis is carried out in combination with the face features, and a face score is calculated; 3, judging the face recognition accuracy according to the face scoring result; 4, extracting voice features of the crowd with low face recognition accuracy; step 5, combining the voice features to form a final voice feature, and matching the final voice feature with a pre-stored voice instruction; and step 6, sending a final instruction. According to the face recognition method, the face score is calculated through the face features, the face score is compared with the standard threshold value, the face information can be captured more comprehensively through the face recognition technology, and therefore the recognition accuracy is improved, and under the condition that the face recognition accuracy is low, judgment is carried out through the voice features, an instruction is sent out, and the recognition accuracy is improved.

The present invention discloses a classroom teaching quality evaluation method based on a large language model, which mainly comprises: based on classroom monitoring video data and teaching system data, using artificial intelligence technology to extract voice, image, and text multimodal features of teachers, students, and classroom environments; constructing a course knowledge graph according to course lesson plans and course information; enhancing the large language model through the constructed course knowledge graph, and realizing intelligent classroom teaching quality evaluation based on the enhanced large language model; the beneficial effects of the present invention are: enhancing the large language model through classroom teaching quality evaluation knowledge and the course knowledge graph, guiding the large language model to generate an evaluation result description that conforms to the classroom teaching quality evaluation facts based on the semantic association between the classroom teaching quality evaluation knowledge and the course knowledge graph; thereby enabling the model to generate a more reasonable description that conforms to the classroom teaching quality evaluation facts.

The embodiment of the invention provides a multi-mode voice wake-up method and device and electronic equipment. The method is applied to the technical field of intelligent voice, and comprises the steps of providing a basis for subsequent accurate processing by acquiring voice, gesture and mouth shape multi-mode input data and adding a timestamp, avoiding a high misjudgment rate of a single voice mode in a complex environment, greatly improving wake-up accuracy, and when a mode conflict exists, improving wake-up accuracy. By identifying the confidence of each modal input data and combining the historical data to make a decision, the possibility of wrong wake-up operation caused by a single modal identification error is reduced, the reliability of wake-up decision is improved, if modal conflicts do not exist, the wake-up scene of the target equipment is accurately determined according to the multi-modal input data, and the wake-up efficiency of the target equipment is improved. Through accurate identification and adaptive processing of different wake-up scenes, the target device can be stably and accurately waken up in various complex environments, and the environmental adaptability of the target device is effectively enhanced.

The invention relates to the technical field of speech enhancement, in particular to a full-band speech enhancement method based on an earphone inertial sensor, which comprises the following steps: performing user registration and limited data acquisition; carrying out segmentation, denoising, dimension reduction and normalization technologies on the collected original data to obtain error-free data; converting inertial sensor data into high-dimensional pronunciation kinematics video features by using a GAN-based method; realizing cross-modal voice frequency domain information enhancement based on a double-flow codec structure; adopting an unsupervised field adaptive technology based on adversarial training to adapt the pre-training model to a new target user; and estimating voice phase information by using a phase denoising network, and reconstructing a time domain voice signal in combination with frequency spectrum information. Unified speech enhancement of full frequency bands is realized, so that an inertia channel and an acoustic channel are self-adaptively complementary in a dynamic environment, and meanwhile, low delay, low power consumption and high quality are taken into account.

The invention provides an intelligent accompanying-oriented emotional personification multi-modal voice interaction large model system, which relates to the field of language processing and is characterized by comprising the following modules: a user interaction module, a multi-modal data fusion module, a database module, a data analysis module and a function processing module, the user interaction module comprises touch interaction, voice interaction, text interaction and local interaction. The intelligent accompanying toy has the advantages that by integrating core technologies such as multi-modal data fusion, hierarchical memory and RAG, dynamic emotion response and edge-cloud collaboration, the problems of emotion interaction templating, lack of role consistency, high response delay, insufficient utilization of complex modals, weak long-term memory ability and the like of an existing intelligent accompanying toy are systematically solved; and the reality sense, the individuation degree and the real-time performance of interaction are remarkably improved, so that high-quality accompanying experience with higher emotional value and immersion is provided for the user.

The invention discloses a multi-mode voice interaction method, device and equipment and a readable storage medium, and relates to the technical field of voice interaction. Comprising the following steps: firstly, acquiring reflected ultrasonic information reflected based on emitted ultrasonic information, and judging whether a target person exists in a target space or not according to the ultrasonic information; if the target person exists in the target space and a voice wake-up instruction is received, orientation information of the voice wake-up instruction is determined based on the microphone array, and a continuous motion track of the target person is determined according to the orientation information and the reflected ultrasonic information; and finally, noise reduction and enhancement are carried out on a voice instruction sent by the target person based on the continuous motion track, and voice interaction is carried out based on the voice instruction after noise reduction and enhancement. The distance information of the azimuth information of the target person is mastered through the reflection of the ultrasonic waves, and then the continuous motion trail of the target person is generated, so that the tracking noise reduction and enhancement of the voice instruction sent to the target person are realized, and the stability of voice interaction can be enhanced without a camera.

The application relates to the technical field of speech recognition, and discloses a cross-modal speech recognition method. The method comprises the following steps: obtaining video information to be analyzed and extracting call audio and visual information therefrom; obtaining a speech frame sequence; performing decoding operation on the speech frame sequence by using a speech recognition model to obtain corresponding text information; performing multiple feature extraction operation on the visual information to obtain a feature sequence of the visual information; obtaining a face information sequence by using a preset visual information extraction model; performing extraction analysis on the face information sequence by using a preset target detection model, so that further optimization is performed; performing decoding operation on the optimized lip sequence by using a trained lip movement conversion model to obtain a candidate word set; and comparing and fusing by using a trained fusion neural model to output final text information. The application can solve the problem that current speech recognition is prone to interference.