79 results about "Speaking style" patented technology

The invention discloses a self-adaptive voiceprint recognition system based on a Mama model. The self-adaptive voiceprint recognition system comprises a voice acquisition module, a voice enhancement module, a voice feature extraction module, a spectrogram enhancement module, a model training module and a model decision module, according to the invention, through well-designed network architecture and algorithm optimization, high identification accuracy is ensured, and good robustness is shown; the system can keep a low equal error rate under various noise conditions and in different speaking styles, which indicates high robustness and accuracy of the system in a voiceprint recognition task; a bidirectional state space model of the Mama module can effectively capture local details and global context of voiceprints, and the recognition capability of the system on complex voice features is enhanced; the TDNN module strengthens the sensitivity of the system to time sequence features, the DenseNet module improves the transmission and fusion of the features through the dense connection of the DenseNet module, and the combination of the TDNN module and the DenseNet module effectively improves the feature extraction fineness of the Mamba module and improves the robustness of the model.

A method and a system for generating human speech audio in a conversation using a trained generative AI model are provided. The method includes receiving a text input representing a portion of the conversation, receiving dialog context associated with the conversation, receiving information representing at least one voice and speaking style of at least one speaker in the conversation, generating the at least one voice and speaking style based on the received information, and generating at least one emotional audio response for the at least one speaker using the at least one voice and speaking style and without retraining the trained generative AI model.

The invention discloses a multi-modal adaptive digital human system based on deep learning, and the system comprises a facial expression extraction module which extracts facial information from a video image through employing a convolutional neural network in deep learning; the speaking style modeling module is used for analyzing and understanding the language mode of the user by using a long-term short-term memory network so as to predict the intention and emotional state of the user; and the personalized recommendation module is combined with collaborative filtering and a content-based recommendation algorithm to deeply mine and learn historical data of a user, and the invention relates to the technical field of a multi-modal adaptive digital human system. The multi-modal adaptive digital human system based on deep learning combines rich sensor information and advanced artificial intelligence technology, has high-precision emotion recognition ability and flexible adaptive dialogue ability, can effectively increase the sense of reality and intimacy in the process of human-computer interaction, and improves the accuracy of human-computer interaction. And the use satisfaction of the user is greatly improved.

The invention provides a voice synthesis method and system based on VITS improvement, and the method comprises the steps: optimizing a text encoder of a VITS model, introducing a large language model, and enabling the emotion, intention and speaking style of an input text to be captured when the text is encoded; a random disturbance item is introduced when the Q value is dynamically planned and solved, the alignment flexibility in the initial training stage is improved, meanwhile, monotonicity constraint is strictly kept, and it is avoided that a suboptimal solution is obtained through convergence too early; a ConvNeXt module is used as a basic backbone network of a decoder, and ISTFT is utilized to efficiently reconstruct a time domain signal, so that waveform up-sampling is realized, redundant calculation of traditional transpose convolution is avoided, and reasoning is accelerated. According to the method, the reasoning speed, the emotion expression ability and the style control flexibility of speech synthesis can be effectively improved, a new solution is provided for cross-language diversified speech synthesis, and a reference is provided for the more efficient and more intelligent development of the speech synthesis technology.

The invention discloses a voice-driven facial animation simulation method based on facial muscle linkage, and belongs to the field of artificial intelligence, and the method comprises the steps: S1, constructing a PPMF encoder; s2, constructing a decoder based on FDCP to decode FP features provided by the PPMF so as to obtain a facial animation; s3, training a 3D face animation framework DCPTalk driven by voice; s4, optimizing the model; and S5, quantitatively evaluating the model. According to the invention, a DCPTalk framework is provided, and the Mouth2Face is provided based on the linkage characteristic of the facial muscle group. Motion of the mouth has strong correlation with voice signals, and can be synthesized with sound track dynamics easily. In order to further enhance the details of the facial movement, the Refine Decoder is adopted to simulate the skin deformation of the surface layer to refine the facial animation. Inherent body characteristics and body characteristics related to facial muscle group movement are embedded into Mouth2Face, a personalized facial muscle control system is constructed, and meanwhile, an external driving signal is modulated by using a speaking style. Qualitative and quantitative experiments and user research show that the DCPTalk is superior to the most advanced method in the prior art.

According to one embodiment, a method, computer system, and computer program product for augmenting a digital audio representation of a voice is provided. The embodiment may include identifying a current voice waveform of a user. The current voice waveform corresponds to captured speech output of the user. The embodiment may include comparing one or more frequency components of the current voice waveform to one or more corresponding frequency components of a baseline voice waveform of the user. In response to determining that at least one of the one or more frequency components of the current waveform fail a threshold degree of match to at least one corresponding frequency component of the baseline voice waveform, the embodiment may include augmenting the captured speech output via a generative artificial intelligence (AI) voice model trained to produce speech which mimics a voice and a speaking style of the user.

To provide an automatic advertisement distribution system utilizing generative AI for providing information on a new commodity or service to a customer by utilizing normally useless time such as holding time of a telephone call to the maximum.SOLUTION: Provided is an automatic advertisement distribution system utilizing generative AI, and the system includes: means for utilizing normally useless time such as holding time of a telephone call to the maximum; means for automatically analyzing the attribute of a customer from voice and the way of speaking; and means for generating a catchy and attractive advertisement message by the generative AI.SELECTED DRAWING: Figure 1

The embodiment of the invention provides a speech synthesis method and device, a storage medium and an electronic device, and the method comprises the steps: inputting an input text and a specified target identity into an audio output model, extracting an identity feature sequence of the target identity through an identity recognition model, extracting a phoneme feature sequence corresponding to the input text through a coding layer of the speech synthesis model, superposing the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text, and inputting the superposed identity feature sequence and the superposed phoneme feature sequence into a variable adapter of the speech synthesis model, and performing duration prediction, alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, outputting target Mel spectrum features corresponding to the input text through a decoding layer of a speech synthesis model, inputting the target Mel spectrum features into a vocoder, and outputting the target Mel spectrum features corresponding to the input text. And outputting a synthetic audio according with the target identity speaking style through the vocoder.

To provide a system capable of achieving support for elderly people living alone and early detection of dementia.SOLUTION: Provided is a system with an AI partner that can be customized according to a user's preferences and interests, the system comprising: means for estimating an emotional state judging from the user's tone of voice, manner of speaking, used words, etc. with help of an emotion engine; means for generating prompt sentences to instruct generation of appropriate contents of conversation according to a current emotional state of the user on the basis of the estimated emotional state of the user; means for generating contents of conversation with the user using the generated prompt sentences and the AI partner; means for replying the generated contents of conversation to the user with voice or text; means for finding out an early symptom of dementia through conversation with the user by analyzing changes, etc. in conversation patterns and memory in a short period of time; and means for transmitting an alert to his / her family or a medical institution if the user shows a symptom that may be dementia.SELECTED DRAWING: Figure 1

A method includes obtaining a reference image and a reference speech, the reference image specifying a head of a target object in the video, and the reference speech specifying a voice of the target object; and generating, based on the reference image and the reference speech, a fusion vector by combining a feature of the head and a feature of the voice. The method further includes generating, based on the fusion vector, a plurality of video frames in a video that represents the target object speaking in a timbre of the reference speech by denoising a plurality of initial frames including noise; and generating the video based on the plurality of video frames. In embodiments of the present disclosure, a video in which a semantic feature and a speaking style of the target object are merged can be generated, and the resolution and quality of the generated video are enhanced.

The invention relates to the technical field of intelligent robots, in particular to a robot self-adaptive interaction method and system. The robot adaptive interaction system comprises a voice data acquisition module, a dialogue voice data construction module and a dialogue text generation model updating module. In the interaction process of the robot and the person, the voice data of the user is obtained, the text data converted from the voice data is processed through the dialogue text generation model, the dialogue text data used for constructing the dialogue voice data is generated, the robot replies based on the dialogue voice data, interaction of the robot is achieved, and the interaction efficiency of the robot is improved. In the dialogue text generation model, semantic feature vectors are extracted based on the speaking style of the user, and the semantic feature vectors are further updated through multiple rounds of dialogues, so that dialogue text data generated through the text data and the standard semantic feature vectors can conform to the speaking habits of the user; and the experience of the user during interaction is improved.

The invention relates to the technical field of artificial intelligence, and provides an interactive dialogue method and device, equipment and a storage medium, and the method comprises the steps: collecting a voice signal inputted by a user, and carrying out the recognition of the voice signal, and obtaining first text content corresponding to the voice signal; extracting voice emotion representation of the voice signal; extracting a text emotion representation of the first text content; fusing the speech emotion representation and the text emotion representation to obtain emotion information; according to the first text content, determining second text content which has a dialogue with the first text content; and outputting dialogue voice according to the emotion information and the second text content. According to the method, the voice emotion representation and the text emotion representation are fused, so that the real emotion state of the user is fully known, and the emotion information matched with the emotion state or the speaking style of the user is obtained. According to the emotional information and the second text content, the dialogue voice matched with the emotional state or the speaking style of the user is output, and the authenticity of the interactive dialogue is improved.

To provide a system for enabling persons who remain following the death of a decedent, to mitigate their feelings of loss and to recover their mental health.SOLUTION: A system includes a learning part, an incorporating part, an instruction part, an interactive part, and a stop part. The learning part generates AI having learned a user's voice, way of speaking, and personality. The incorporating part incorporates the AI generated by the learning part in a tablet terminal. The instruction part instructs delivery of a tablet terminal to a person specified by a user, after the death of the user. The interactive part interacts with a person specified by the user, by using the AI incorporated in the tablet terminal. The stop part stops services according to AI after the lapse of a predetermined period after the death of the user.SELECTED DRAWING: Figure 1

To provide a shooting application that enables video shooting in a specified format and allows easy acquisition of user information on a mobile device such as a smartphone in order to construct a virtual human.SOLUTION: A photographing application for acquiring voice characteristics of a user's face and speaking style has a video shooting function for shooting a video in a first designated format including at least the user's face direction, facial expression, shooting time, and whether a dialogue is read, and a dialogue reading video shooting function for shooting a dialogue reading video in a second designated format including at least the user's face direction, facial expression, number of sentences, and whether a pause is read.SELECTED DRAWING: Figure 2

To solve the problem of presentation support.SOLUTION: A system for presentation support includes: means by which a user inputs presentation materials and voice data; means by which generative AI provides suitable feedback; and means for inputting voice data converted to text and sound volume or feeling data converted to a vector to return feedback on how to talk.SELECTED DRAWING: Figure 1

The invention provides an ASV system risk assessment method and system based on multi-dimensional pronunciation characterization decoupling and fusion, and the method comprises the steps: obtaining a reference audio of a target user, and carrying out the decoupling extraction of a volume feature vector, a pitch feature vector, and a speaking style feature vector from the reference audio through a multi-dimensional pronunciation feature extraction network; a text to be verified is converted into a phoneme sequence, the phoneme sequence is input into a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, and three pronunciation feature vectors are injected into the network through an adaptive instance normalization mechanism to dynamically modulate a prediction process. Generating a frame-level dynamic pronunciation feature sequence containing time sequence rhythm change; inputting a VAE-GAN synthesis trunk, re-injecting the three pronunciation feature vectors through an adaptive instance normalization mechanism, and generating a test voice sample consistent with the voiceprint characteristics of the target user; and initiating an identity verification query for the ASV system, and calculating safety indexes of different user groups according to a verification result so as to evaluate the risk of the ASV system.

The invention relates to the technical field of three-dimensional face animation generation, in particular to a voice-driven dynamic three-dimensional face and dynamic texture generation method and system, and the method comprises the following steps: S1, training two independent auto-encoders to respectively learn change primitives of face geometry and wrinkles, the facial motion map and the wrinkle map are compressed to low-dimensional implicit features; s2, based on implicit features zw and zf generated by an encoder, training a collaborative generation network based on a diffusion model, and generating clean implicit features from noise input under the guidance of audio features; and S3, constructing an implicit facial motion feature by using the speaking style axis feature pf, i of the style i, and constructing an implicit texture change feature by using the wrinkle style axis feature pw, j of the style j. The method can effectively solve the problems that an existing voice-driven face animation method cannot generate dynamic texture maps which are consistent and coordinated with face movement at the same time, the sense of reality of the face animation is reduced due to the lack of important face details, even the terrorist valley effect is caused, and the animation effect is affected.

To provide a real-time distribution system that is capable of enhancing communication between a distributor and a viewer at low costs and enables the distributor to obtain a benefit.SOLUTION: A real-time live distribution system for performing real-time live distribution of an idle talk dialog content toward a plurality of viewer terminals comprises a distributor terminal 1 including: an extraction section 11 having a virtual human for extracting viewer comments input from the plurality of viewer terminals in real time; an analysis section 12 for performing text analysis on the extracted viewer comments; a reading section 13 for reading a text using voice synthesis on the basis of a text analysis result; a feeling parameter setting section 14 for setting a virtual human feeling parameter for categorizing the viewer comments, which are at least a positive comment or negative comment, to determine the positive comment as a plus comment and the negative comment as a minus comment; and a change section 15 for changing a way of speaking according to the set feeling parameter.SELECTED DRAWING: Figure 2

The present disclosure provides a video generation method, apparatus, device, and storage medium, which relate to the fields of artificial intelligence technology, particularly computer vision, deep learning, and other technical fields, and can be applied to scenarios such as artificial intelligence-based content generation. A specific implementation scheme is as follows: a voice file and a face video to be driven are input into a pre-trained first model, and the first model outputs a three-dimensional face mesh sequence; wherein the three-dimensional face mesh sequence corresponds to the voice features of the voice file, and corresponds to the facial features and speaking style features of the face video to be driven; based on the three-dimensional face mesh sequence and the face video to be driven, a voice-driven face video matching the voice file is generated.

The invention discloses an audio driven facial animation supporting varying identity and speaking styles. In various examples, systems and methods are disclosed for animating a virtual or digital actor or image using audio-driven animation. The system may identify an animation of the grid corresponding to the audio data and the indication of the speaking style. The system may generate a plurality of vertex increments using animations and neutral gestures of the grid. The system may update the machine learning model using the plurality of vertex increments, the audio data, and the indication of the speaking style to generate an output vertex increment for the grid given an input speaking style and input audio data.

An object of a system according to an embodiment is to detect an emotion of a user and provide appropriate advice on the basis of the detected emotion.SOLUTION: A system according to an embodiment includes a smart emotion detection speaker, an emotion detection AI, and a mentoring generation AI. The smart speaker detects emotions from the user's voice and speech. The emotion detection AI generates words based on emotions. The mentoring generation AI listens to the user's distress and provides advice.SELECTED DRAWING: Figure 1

A system is provided.SOLUTION: A system, comprising: means for selecting a person with whom a user wishes to interact; means for generating a AI model to mimic the person's thought patterns and speech; means for receiving a user question and generating a response based on the AI model; and means for displaying the generated response to the user.SELECTED DRAWING: Figure 1

The present disclosure relates to a method and device for generating a speaking style, an electronic device and a storage medium. The method comprises: fitting target style feature attributes based on multiple style feature attributes, determining fitting coefficients of each style feature attribute; determining a target style feature vector according to the fitting coefficients of each style feature attribute and multiple style feature vectors, the multiple style feature vectors corresponding to the multiple style feature attributes one by one; inputting the target style feature vector into a speaking style model to output target speaking style parameters, the speaking style model being obtained based on a framework of training a speaking style model based on the multiple style feature vectors; and generating a target speaking style based on the target speaking style parameters. The method can realize fast migration of the speaking style and improve the generation efficiency of the speaking style.

Several examples disclose systems and methods for animating virtual or digital performers or avatars using audio-driven animation. A system can identify an animation for a mesh corresponding to audio data and a specified speech style. Using the animation and a neutral pose for the mesh, the system can generate a variety of vertex deltas. Based on this variety of vertex deltas, the audio data, and the specified speech style, the system can update a machine learning model to generate output vertex deltas for the mesh, taking into account an input speech style and input audio data.

A system is provided.SOLUTION: Means for recording video and audio data of a face-to-face contact, means for transmitting the recorded video and audio data to a server, means for performing noise removal and text conversion on the transmitted video and audio data, means for analyzing wording and conversation content using a natural language processing technique on text data, means for analyzing a manner of speaking and a tone of voice from the audio data, means for analyzing facial expression and body language from the video data, and means for digitizing each analyzed feature, A system comprising: means for scoring; means for training a learning model using interview data of existing employees; means for evaluating new interview data using the learning model; and means for displaying evaluation results and providing feedback to a user.SELECTED DRAWING: Figure 1

To provide means for reproducing a conversation with a deceased person and speech synthesis technique.SOLUTION: There is provided a system comprising: means for learning a manner of speaking and a mode of expression of a deceased person, and reproducing the manner of speaking and the mode of expression; and means for executing speech synthesis technique using a generation system AI.SELECTED DRAWING: Figure 1

This invention relates to a method for generating facial expression animations of a voice-driven 3D virtual avatar, comprising: inputting a speech segment and converting it into audio features; an encoder module encoding the audio features extracted by the audio processing module, a static 3D head mesh template, and the input one-hot encoding of the speaking style, encoding them into the latent space, and concatenating them; processing the concatenated feature vector sequence through a bidirectional Mamba module, capturing contextual information for each feature vector position, and outputting a feature vector sequence after information exchange of the same dimension; a decoder decoding the feature vector sequence output by the bidirectional Mamba module to obtain the vertex position offsets of each frame of the face mesh, adding them to the static 3D head mesh template to obtain a 3D head mesh sequence; and using the audio-head mesh sequence dataset to perform end-to-end training of the entire network model.

The invention provides a voice conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the extraction of content features and fundamental frequency features of a source audio, and carrying out the extraction of speaker features of a target audio; inputting the content features, the fundamental frequency features and the speaker features into a voice conversion model for joint modeling processing, linear mapping processing and waveform reconstruction processing to generate a voice waveform; performing one-dimensional deep separation convolution processing and multi-receptive-field fusion processing on the voice waveform based on a vocoder in the voice conversion model to generate converted voice audio; wherein the converted voice audio shows that the speaking style of the source audio is replaced by the speaker style of the target audio. The voice conversion model is used for voice conversion, so that the timbre consistency and semantic retention capability of cross-language voice conversion are improved, and the naturalness and definition of reconstructed voice are improved.

The invention relates to a voice-driven three-dimensional virtual image expression animation generation method, which comprises the following steps of: inputting a voice segment, and converting the voice segment into an audio feature; the encoder module encodes the audio features extracted by the audio processing module, the static three-dimensional head grid template and the input speaking style one-hot code, encodes the audio features, the static three-dimensional head grid template and the input speaking style one-hot code into a hidden space, and splices the audio features, the static three-dimensional head grid template and the input Processing the spliced feature vector sequence through a bidirectional Mama module, capturing context information for the feature vector of each position, and outputting the feature vector sequence after information exchange of the same dimension; the decoder decodes the feature vector sequence output by the bidirectional Mama module to obtain the offset of each vertex position of each frame of face grid, and adds the offset to a static three-dimensional head grid template to obtain a three-dimensional head grid sequence; and performing end-to-end training on the whole network model by using the audio-head grid sequence data set.

We provide the system. [Solution] A means for a person with specialized knowledge to input that knowledge and speaking style as information, A method for training an AI model using natural language processing technology based on that input information, A means of storing and making available trained AI models on the platform, A means for users to search for AI models through the platform and begin interacting with them, A means by which an AI model generates a response to that AC input and sends it back to the user, A means of providing responses as an interactive learning experience and enhancing immersion through visual and auditory means, A system that includes this.