60 results about "Speaking style" patented technology

A method and a system for generating human speech audio in a conversation using a trained generative AI model are provided. The method includes receiving a text input representing a portion of the conversation, receiving dialog context associated with the conversation, receiving information representing at least one voice and speaking style of at least one speaker in the conversation, generating the at least one voice and speaking style based on the received information, and generating at least one emotional audio response for the at least one speaker using the at least one voice and speaking style and without retraining the trained generative AI model.

The invention provides a voice synthesis method and system based on VITS improvement, and the method comprises the steps: optimizing a text encoder of a VITS model, introducing a large language model, and enabling the emotion, intention and speaking style of an input text to be captured when the text is encoded; a random disturbance item is introduced when the Q value is dynamically planned and solved, the alignment flexibility in the initial training stage is improved, meanwhile, monotonicity constraint is strictly kept, and it is avoided that a suboptimal solution is obtained through convergence too early; a ConvNeXt module is used as a basic backbone network of a decoder, and ISTFT is utilized to efficiently reconstruct a time domain signal, so that waveform up-sampling is realized, redundant calculation of traditional transpose convolution is avoided, and reasoning is accelerated. According to the method, the reasoning speed, the emotion expression ability and the style control flexibility of speech synthesis can be effectively improved, a new solution is provided for cross-language diversified speech synthesis, and a reference is provided for the more efficient and more intelligent development of the speech synthesis technology.

A method includes obtaining a reference image and a reference speech, the reference image specifying a head of a target object in the video, and the reference speech specifying a voice of the target object; and generating, based on the reference image and the reference speech, a fusion vector by combining a feature of the head and a feature of the voice. The method further includes generating, based on the fusion vector, a plurality of video frames in a video that represents the target object speaking in a timbre of the reference speech by denoising a plurality of initial frames including noise; and generating the video based on the plurality of video frames. In embodiments of the present disclosure, a video in which a semantic feature and a speaking style of the target object are merged can be generated, and the resolution and quality of the generated video are enhanced.

The invention relates to the technical field of intelligent robots, in particular to a robot self-adaptive interaction method and system. The robot adaptive interaction system comprises a voice data acquisition module, a dialogue voice data construction module and a dialogue text generation model updating module. In the interaction process of the robot and the person, the voice data of the user is obtained, the text data converted from the voice data is processed through the dialogue text generation model, the dialogue text data used for constructing the dialogue voice data is generated, the robot replies based on the dialogue voice data, interaction of the robot is achieved, and the interaction efficiency of the robot is improved. In the dialogue text generation model, semantic feature vectors are extracted based on the speaking style of the user, and the semantic feature vectors are further updated through multiple rounds of dialogues, so that dialogue text data generated through the text data and the standard semantic feature vectors can conform to the speaking habits of the user; and the experience of the user during interaction is improved.

The invention provides an ASV system risk assessment method and system based on multi-dimensional pronunciation characterization decoupling and fusion, and the method comprises the steps: obtaining a reference audio of a target user, and carrying out the decoupling extraction of a volume feature vector, a pitch feature vector, and a speaking style feature vector from the reference audio through a multi-dimensional pronunciation feature extraction network; a text to be verified is converted into a phoneme sequence, the phoneme sequence is input into a pronunciation feature prediction network for predicting a frame-level dynamic pronunciation feature sequence based on the phoneme sequence, and three pronunciation feature vectors are injected into the network through an adaptive instance normalization mechanism to dynamically modulate a prediction process. Generating a frame-level dynamic pronunciation feature sequence containing time sequence rhythm change; inputting a VAE-GAN synthesis trunk, re-injecting the three pronunciation feature vectors through an adaptive instance normalization mechanism, and generating a test voice sample consistent with the voiceprint characteristics of the target user; and initiating an identity verification query for the ASV system, and calculating safety indexes of different user groups according to a verification result so as to evaluate the risk of the ASV system.

The invention relates to the technical field of three-dimensional face animation generation, in particular to a voice-driven dynamic three-dimensional face and dynamic texture generation method and system, and the method comprises the following steps: S1, training two independent auto-encoders to respectively learn change primitives of face geometry and wrinkles, the facial motion map and the wrinkle map are compressed to low-dimensional implicit features; s2, based on implicit features zw and zf generated by an encoder, training a collaborative generation network based on a diffusion model, and generating clean implicit features from noise input under the guidance of audio features; and S3, constructing an implicit facial motion feature by using the speaking style axis feature pf, i of the style i, and constructing an implicit texture change feature by using the wrinkle style axis feature pw, j of the style j. The method can effectively solve the problems that an existing voice-driven face animation method cannot generate dynamic texture maps which are consistent and coordinated with face movement at the same time, the sense of reality of the face animation is reduced due to the lack of important face details, even the terrorist valley effect is caused, and the animation effect is affected.

To provide a real-time distribution system that is capable of enhancing communication between a distributor and a viewer at low costs and enables the distributor to obtain a benefit.SOLUTION: A real-time live distribution system for performing real-time live distribution of an idle talk dialog content toward a plurality of viewer terminals comprises a distributor terminal 1 including: an extraction section 11 having a virtual human for extracting viewer comments input from the plurality of viewer terminals in real time; an analysis section 12 for performing text analysis on the extracted viewer comments; a reading section 13 for reading a text using voice synthesis on the basis of a text analysis result; a feeling parameter setting section 14 for setting a virtual human feeling parameter for categorizing the viewer comments, which are at least a positive comment or negative comment, to determine the positive comment as a plus comment and the negative comment as a minus comment; and a change section 15 for changing a way of speaking according to the set feeling parameter.SELECTED DRAWING: Figure 2

The present disclosure provides a video generation method, apparatus, device, and storage medium, which relate to the fields of artificial intelligence technology, particularly computer vision, deep learning, and other technical fields, and can be applied to scenarios such as artificial intelligence-based content generation. A specific implementation scheme is as follows: a voice file and a face video to be driven are input into a pre-trained first model, and the first model outputs a three-dimensional face mesh sequence; wherein the three-dimensional face mesh sequence corresponds to the voice features of the voice file, and corresponds to the facial features and speaking style features of the face video to be driven; based on the three-dimensional face mesh sequence and the face video to be driven, a voice-driven face video matching the voice file is generated.

The invention discloses an audio driven facial animation supporting varying identity and speaking styles. In various examples, systems and methods are disclosed for animating a virtual or digital actor or image using audio-driven animation. The system may identify an animation of the grid corresponding to the audio data and the indication of the speaking style. The system may generate a plurality of vertex increments using animations and neutral gestures of the grid. The system may update the machine learning model using the plurality of vertex increments, the audio data, and the indication of the speaking style to generate an output vertex increment for the grid given an input speaking style and input audio data.

An object of a system according to an embodiment is to detect an emotion of a user and provide appropriate advice on the basis of the detected emotion.SOLUTION: A system according to an embodiment includes a smart emotion detection speaker, an emotion detection AI, and a mentoring generation AI. The smart speaker detects emotions from the user's voice and speech. The emotion detection AI generates words based on emotions. The mentoring generation AI listens to the user's distress and provides advice.SELECTED DRAWING: Figure 1

A system is provided.SOLUTION: A system, comprising: means for selecting a person with whom a user wishes to interact; means for generating a AI model to mimic the person's thought patterns and speech; means for receiving a user question and generating a response based on the AI model; and means for displaying the generated response to the user.SELECTED DRAWING: Figure 1

The present disclosure relates to a method and device for generating a speaking style, an electronic device and a storage medium. The method comprises: fitting target style feature attributes based on multiple style feature attributes, determining fitting coefficients of each style feature attribute; determining a target style feature vector according to the fitting coefficients of each style feature attribute and multiple style feature vectors, the multiple style feature vectors corresponding to the multiple style feature attributes one by one; inputting the target style feature vector into a speaking style model to output target speaking style parameters, the speaking style model being obtained based on a framework of training a speaking style model based on the multiple style feature vectors; and generating a target speaking style based on the target speaking style parameters. The method can realize fast migration of the speaking style and improve the generation efficiency of the speaking style.

Several examples disclose systems and methods for animating virtual or digital performers or avatars using audio-driven animation. A system can identify an animation for a mesh corresponding to audio data and a specified speech style. Using the animation and a neutral pose for the mesh, the system can generate a variety of vertex deltas. Based on this variety of vertex deltas, the audio data, and the specified speech style, the system can update a machine learning model to generate output vertex deltas for the mesh, taking into account an input speech style and input audio data.

A system is provided.SOLUTION: Means for recording video and audio data of a face-to-face contact, means for transmitting the recorded video and audio data to a server, means for performing noise removal and text conversion on the transmitted video and audio data, means for analyzing wording and conversation content using a natural language processing technique on text data, means for analyzing a manner of speaking and a tone of voice from the audio data, means for analyzing facial expression and body language from the video data, and means for digitizing each analyzed feature, A system comprising: means for scoring; means for training a learning model using interview data of existing employees; means for evaluating new interview data using the learning model; and means for displaying evaluation results and providing feedback to a user.SELECTED DRAWING: Figure 1

This invention relates to a method for generating facial expression animations of a voice-driven 3D virtual avatar, comprising: inputting a speech segment and converting it into audio features; an encoder module encoding the audio features extracted by the audio processing module, a static 3D head mesh template, and the input one-hot encoding of the speaking style, encoding them into the latent space, and concatenating them; processing the concatenated feature vector sequence through a bidirectional Mamba module, capturing contextual information for each feature vector position, and outputting a feature vector sequence after information exchange of the same dimension; a decoder decoding the feature vector sequence output by the bidirectional Mamba module to obtain the vertex position offsets of each frame of the face mesh, adding them to the static 3D head mesh template to obtain a 3D head mesh sequence; and using the audio-head mesh sequence dataset to perform end-to-end training of the entire network model.

The invention provides a voice conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the extraction of content features and fundamental frequency features of a source audio, and carrying out the extraction of speaker features of a target audio; inputting the content features, the fundamental frequency features and the speaker features into a voice conversion model for joint modeling processing, linear mapping processing and waveform reconstruction processing to generate a voice waveform; performing one-dimensional deep separation convolution processing and multi-receptive-field fusion processing on the voice waveform based on a vocoder in the voice conversion model to generate converted voice audio; wherein the converted voice audio shows that the speaking style of the source audio is replaced by the speaker style of the target audio. The voice conversion model is used for voice conversion, so that the timbre consistency and semantic retention capability of cross-language voice conversion are improved, and the naturalness and definition of reconstructed voice are improved.

The invention relates to a voice-driven three-dimensional virtual image expression animation generation method, which comprises the following steps of: inputting a voice segment, and converting the voice segment into an audio feature; the encoder module encodes the audio features extracted by the audio processing module, the static three-dimensional head grid template and the input speaking style one-hot code, encodes the audio features, the static three-dimensional head grid template and the input speaking style one-hot code into a hidden space, and splices the audio features, the static three-dimensional head grid template and the input Processing the spliced feature vector sequence through a bidirectional Mama module, capturing context information for the feature vector of each position, and outputting the feature vector sequence after information exchange of the same dimension; the decoder decodes the feature vector sequence output by the bidirectional Mama module to obtain the offset of each vertex position of each frame of face grid, and adds the offset to a static three-dimensional head grid template to obtain a three-dimensional head grid sequence; and performing end-to-end training on the whole network model by using the audio-head grid sequence data set.

We provide the system. [Solution] A means for a person with specialized knowledge to input that knowledge and speaking style as information, A method for training an AI model using natural language processing technology based on that input information, A means of storing and making available trained AI models on the platform, A means for users to search for AI models through the platform and begin interacting with them, A means by which an AI model generates a response to that AC input and sends it back to the user, A means of providing responses as an interactive learning experience and enhancing immersion through visual and auditory means, A system that includes this.

The invention discloses a virtual image construction method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring target text data and first video data; performing emotion recognition processing on the target text data to obtain a corresponding expression tag; performing voice conversion processing on the target text data based on the expression label to obtain target voice data; driving the lip of the target user in the first video data to act by taking the target voice data as a driving signal to obtain second video data; extracting a first mouth opening and closing feature in the first video data, and determining a first speaking style feature index; adjusting the general expression feature data based on the first speaking style feature index and the general speaking style feature index to obtain expression feature data corresponding to the target user; and performing expression editing processing on the second video data by using the expression feature data to obtain the target video data, thereby enhancing the reality sense of the virtual image and the immersion sense of the user.

An object of the system according to the embodiment is to reproduce the voice and the way of speaking of the deceased and to reestablish emotional connection.SOLUTION: A system includes a collection unit, a learning unit, a reception unit, a generation unit, and a reproduction unit. The collection unit collects video and audio data of the deceased. The learning unit learns the data collected by the collection unit and analyzes the voice and the way of speaking of the deceased. The receiving unit receives an input from a user. The generation unit generates new words in the voice of the deceased based on the input accepted by the acceptance unit. The reproduction unit reproduces the words generated by the generation unit.SELECTED DRAWING: Figure 1

Control over speaking style of a text-to-speech (TTS) system is provided without necessarily requiring that the training of the TTS conversion process (e.g., the ANN used for the conversion) take into account the speaking styles of the training data. For example, the TTS system may allow adjustment of characteristics of speaking styles, such as, speed, perceivable degree of “kindness”, average pitch, pitch variation, and duration of pauses. In some examples, a voice designer may have a number of independent controls that vary corresponding characteristics without necessarily varying others. Once the designer has configured a desired overall speaking style based on those controllable characteristics, the TTS system can be configured to use that speaking style for deployments of the TTS system. For example, the TTS system may be used for audio output in a voice assistant, for instance, for an in-vehicle voice assistant.

A system is provided.SOLUTION: A system comprising: a default setting unit configured to allow a user to register; a unit configured to collect and store conversation data with the user; a unit configured to analyze the collected conversation data and learn a manner of speaking and a context of the user; a unit configured to train a user-specific model based on the learned data; a unit configured to generate a reply of the conversation using the generated user-specific model; and a unit configured to transmit the generated reply to a user terminal.SELECTED DRAWING: Figure 1

The invention discloses a multi-party dialogue model training method, a related device thereof and a multi-party dialogue system, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a multi-party dialogue sample, and a dialogue context sample of a first speaker in a first dialogue of multi-party dialogues; based on the speaking of each speaker in the multi-party dialogue sample, performing comparative learning on a preset encoder to obtain a speaking style perception model; namely, the model can learn the speaking styles of different users without depending on the annotation information through a comparative learning mode; the method comprises the following steps: performing iterative training on a speaking style perception model based on a dialogue context sample to obtain a multi-party dialogue model, and learning how to generate a reply according to a context on the basis of having a speaking style perception capability, so that the multi-party dialogue model can respond to speaks of different speakers in different speaking styles, and the user experience is improved. Personalized communication with the participants is realized, the reply quality and continuity are improved, and the dialogue satisfaction degree of the user is improved.

A method includes obtaining a reference image and a reference speech, the reference image specifying a head of a target object in the video, and the reference speech specifying a voice of the target object; and generating, based on the reference image and the reference speech, a fusion vector by combining a feature of the head and a feature of the voice. The method further includes generating, based on the fusion vector, a plurality of video frames in a video that represents the target object speaking in a timbre of the reference speech by denoising a plurality of initial frames including noise; and generating the video based on the plurality of video frames. In embodiments of the present disclosure, a video in which a semantic feature and a speaking style of the target object are merged can be generated, and the resolution and quality of the generated video are enhanced.

The system according to the embodiment aims to reproduce the speaking style and knowledge of great people, and to provide the user with the feeling that they are having a conversation with a great person. [Solution] A system according to an embodiment includes a collection unit, an analysis unit, a reproduction unit, a reception unit, and an answering unit. The collection unit collects voice data. The analysis unit analyzes the data collected by the collection unit. The reproduction unit reproduces the speaking style and knowledge of the great person based on the data analyzed by the analysis unit. The reception unit accepts input from a user. The answering unit answers like the great person based on the input accepted by the reception unit.

The invention relates to the technical field of artificial intelligence, and discloses an AI interview method, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining an interview text; determining a first speech speed and a first pause frequency of the candidate according to the interview text; the first pause frequency is used for describing the pause frequency in the process that the candidate inputs the interview audio corresponding to the interview text; determining the semantic integrity probability of the interview text; determining the probability that the candidate does not continue speaking according to the first speech speed, the first pause frequency and the semantic integrity probability; and determining whether to continuously ask questions according to the probability that the candidates do not continuously speak. Therefore, the opportunity of continuing to ask questions by the AI interviewer can be adaptively adjusted according to the speaking style of the candidate, so that the AI interviewer takes over a conversation at a reasonable opportunity, and the interview experience feeling of the candidate is improved.

This application provides a model update method and apparatus, a speech conversion method, device, and storage medium, belonging to the field of financial technology. The method includes: acquiring sample speech data; inputting the sample speech data into a neural network model; encoding the sample speech data through an encoding network to obtain an initial audio feature vector; indexing the initial audio feature vector based on a preset codebook to obtain an audio frame index; extracting phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; performing speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector; decoding the sample audio embedding vector and the speech style embedding vector through a decoding network to obtain synthesized speech data; and updating the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech conversion model. This application can improve the accuracy of the model in speech conversion.

An information processing system according to the present disclosure comprises: a reception unit that receives, from a user, subjective evaluation information relating to speech evaluation, and designation of a speaking style; a generation unit that, on the basis of the subjective evaluation information and the designation of the speaking style, converts a low-dimensional embedding vector corresponding to the speaking style adjusted by the user to generate a high-dimensional speaking style embedding vector; and a speech synthesis unit that synthesizes speech from the speaking style embedding vector and text information defining speech to be synthesized.

The application provides a controllable text-to-speech method based on decoupled multi-modal cues and chain-based guidance, which comprises the following steps: constructing a unified multi-modal style encoder for mapping style cues from reference audio or descriptive text to a shared embedding space to generate style-conditioned embeddings, and training the multi-modal style encoder; constructing a text-to-speech model based on a latent diffusion transformer, which takes the latent representation of mel-spectrogram as the processing object and receives content text, speaker prosody and speaking style as input conditions; in the inference stage of the text-to-speech model, a chain-based classifier-free guidance mechanism is adopted to realize independent and continuous control of the content, prosody and style of the generated speech by independently adjusting the guidance strength of the content, prosody and style-conditioned embeddings. The independent and continuous control of the content, prosody and style of the generated speech is realized. The training stability and convergence speed of the model are significantly improved.

To provide an audio-driven three-dimensional facial animation model generation method and a generation apparatus, and an apparatus.SOLUTION: A generation method includes the steps of: acquiring sample data including sample audio data, sample speaking style data, and a sample blend deformation value; performing feature extraction on the sample audio data to obtain a sample audio feature; performing convolution on the sample audio feature based on a to-be-trained audio-driven three-dimensional facial animation model to obtain an initial audio feature, and performing encoding on the sample speaking style data based on the to-be-trained audio-driven three-dimensional facial animation model to obtain a sample speaking style feature; performing encoding on the initial audio feature and the sample speaking style feature based on the to-be-trained audio-driven three-dimensional facial animation model, to obtain an output blend deformation value; and performing calculation on the sample blend deformation value and the output blend deformation value to obtain a loss function value.SELECTED DRAWING: Figure 2A