802 results about "Synthesis methods" patented technology

The invention relates to the technical field of voice processing, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a cross-language voice migration synthesis method, device, equipment and medium. And training an acoustic model by adopting a hierarchical self-adaptive fine tuning strategy, fusing the phoneme sequence for reasoning and the tone mark to generate a representation sequence, and finally synthesizing a target language speech signal. According to the method, a sharing and separation parallel cross-language modeling structure is constructed, the accuracy of phoneme and tone modeling in a low-resource language is effectively improved, the target language acoustic model has higher generalization ability and migration efficiency in combination with a multi-stage self-adaptive fine tuning and training data enhancement strategy, and the method is suitable for being applied to the field of multi-language modeling. And finally, synchronous improvement of the voice naturalness and the tone fidelity is realized.

The invention relates to a speech synthesis method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source language dubbing audio corresponding to a source language subtitle text of a source video and a target language subtitle text translated by the source language subtitle text; extracting audio emotion features from the source language dubbing audio by using an emotion extractor, wherein the audio emotion features represent emotions expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence by using a text encoder to obtain text encoding features; fusing the audio emotion features with the text coding features to obtain emotion text features; and generating a target language audio based on the emotion text features by using a decoder, the target language audio being used as a dubbing audio of the source video in the target language. Therefore, the high-quality target language audio with the source dubbing audio emotion can be automatically and efficiently generated, the cost is low, and the efficiency is high.

The invention relates to a multi-agent-based material performance prediction and synthesis system, and the system comprises a multi-agent data enhancement module which is configured to be used for firstly disassembling a complex problem into a plurality of subtasks, and then constructing a fine tuning data set comprising Sub-CoQ question and answer pairs by starting multi-source parallel retrieval; the multi-expert debate module is configured to be used for simulating decision conflicts of different roles in material engineering and generating a direct preference optimization DPO data set through debate; the training and verification module is configured to be used for training and verifying a large model MatMind in the field of materials by utilizing supervised fine tuning SFT and reinforcement learning RLHF based on the fine tuning data set and the DPO data set; and the material performance prediction and synthesis module is configured to be used for realizing intelligent recommendation of a material performance prediction and synthesis process by importing input parameters into the large model MatMind.

The invention provides a voice replication and low-delay streaming voice synthesis method and system based on an ultra-short sample, relates to the technical field of artificial intelligence, and is suitable for intelligent interaction, outbound service and multi-mode communication scenes. Deep personalized customization of intelligent voice interaction and real-time generation of ultra-low delay are realized, and bidirectional streaming interaction of a system level is supported, so that the fluency and response speed of dialogues are improved; an ultra-short sample sound duplicating module specially designed for processing ultra-short audio samples and a speech synthesis engine with ultra-low delay and bidirectional streaming output capability are integrated, and the capabilities of the ultra-short sample sound duplicating module and the speech synthesis engine are applied to a real-time and interactive intelligent speech interaction process to form a complete and efficient solution. The industrial pain point is directly solved in a targeted manner, and the method has important commercial application value.

The invention discloses a high-definition video mouth shape automatic synthesis method driven by a phoneme time axis, and relates to the technical field of electrical digital data processing, and the method comprises the steps: 1, obtaining an input text or voice, employing a fixed phoneme set and a deterministic pronunciation rule to generate a phoneme sequence and a corresponding starting and ending time point, and building the phoneme time axis; outputting the phoneme time axis, the frame index mapping table and the target frame set; 2, on the basis of a fixed and limited mouth shape dictionary and a pronunciation part rule, three complementary derivation chains are established at the same time, and a mouth shape key frame sequence meeting phase continuity and boundary traceability is obtained; and step 3, extracting three-dimensional face feature points and a head posture track based on the original video, and outputting a video mouth shape synthesis result which is consistent with the resolution of the original video and is synchronous with a phoneme time axis according to the posture uniformization mouth shape sequence. According to the invention, the accuracy and reality of mouth shape video generation are obviously improved.

The invention discloses an arbitrary digital signal synthesis method and system. The method comprises the following steps: analyzing a target parameter of a digital signal to be synthesized; generating a first multi-path baseband waveform by using a digital direct frequency synthesizer according to a modulation rate parameter in the target parameters, and reading pre-stored noise data to generate a second multi-path baseband waveform; selecting a plurality of appointed baseband waveforms, and modulating the plurality of baseband waveforms by using carrier frequency and bandwidth parameters in target parameters to obtain a plurality of instantaneous frequency curves; and integrating the multiple instantaneous frequency curves to obtain a phase stepping curve of each path, selecting a phase modulation curve according to a phase modulation type in the target parameters, and performing digital direct frequency synthesis on the phase stepping curve and the phase modulation curve of each path to generate multiple parallel digital synthesis signals. According to the invention, the synthesis process of the digital signals of different modulation types is uniformly decomposed, so that the coupling degree of the module and the complexity of the system are greatly reduced, and meanwhile, the resource consumption is reduced.

The invention relates to the technical field of speech synthesis, can be applied to business system platforms such as financial science and technology, medical health and the like, and discloses a layered coding and decoding speech synthesis method, device, equipment and medium, and the method comprises the following steps: carrying out semantic feature extraction on a pre-acquired original text to obtain text semantic features; performing prosodic feature extraction on the original text to obtain a text prosodic feature; performing feature fusion compression on the text semantic features and the text rhythm features to obtain fusion compression features; performing low-frequency decoding on the fusion compression features to obtain low-frequency decoding features; performing high-frequency decoding on the fusion compression feature to obtain a high-frequency decoding feature; and combining the low-frequency decoding features and the high-frequency decoding features, and converting the combined features into target synthetic speech. Through high and low frequency decoding, richer and more accurate voice details are restored, so that the synthesized voice is listened more naturally and smoothly, and the credibility and fluency of the voice are improved.

The present disclosure provides a speech synthesis model training method. The method can comprise: acquiring initial training data; selecting a plurality of first text segments, second text segments, and third text segments from coherent text, and acquiring first audio segments, second audio segments, and third audio segments from coherent audio; detecting whether the plurality of audio segments and text segments meet a stitching condition; on the basis of the text sequence, stitching, by alternatively arranging text and audio, the plurality of text segments and audio segments meeting the stitching condition to generate combined training data, so as to generate a combined training data set; on the basis of the combined training data set, training the initial speech generation model to obtain a trained speech synthesis model, wherein during the training, prosodic, tonal, and / or emotional features from the second text segments and / or the third text segments are extracted. Further disclosed is a speech synthesis method. The present disclosure allows for use of the context of the text and audio to realize text comprehension to capture prosody, tonality, and emotion in the context.

The embodiment of the invention provides a voice synthesis method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining multi-modal data which comprises a voice signal, a user instruction, a historical interaction log and bus data; determining model input parameters according to the multi-modal data; according to the model input parameters, the static corpus and a preset large model, generating a personalized script, the preset large model being used for dynamically generating the personalized script in combination with the model input parameters and the static corpus; and synthesizing the broadcast audio according to the personalized script. According to the invention, the technical problem that the voice synthesis technology in the related technology cannot adapt to the personalized demands of the user is solved.

The invention provides a speech synthesis method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is obtained by performing multi-stage prediction task training according to a phoneme sequence and a discrete acoustic feature sequence corresponding to the training text, and a multi-stage prediction task comprises a discrete acoustic feature sequence prediction task and a Mel spectrum feature prediction task. The quality of synthesized speech can be improved.

To provide a control device for a magnetic bearing that enables mode separation of a natural vibration mode with high precision and can design a stable controller with the degree of the controller made small.SOLUTION: With respect to a model for a rotary body controlled by radial magnetic bearings 1, 2, a status space model is structured by excluding a controller which controls a plant from a mode composition model having parameters of a mass matrix, a rigidity matrix, and an attenuation matrix identified and adjusted by a parameter estimation part, and a mode separation part 61 on the input side of controllers 1, 2 inputs a measurement signal of a displacement sensor from the radial magnetic bearings 1, 2 to perform mode separation with a mode separation coefficient. A mode composition part 62 on the output side of the controllers 1, 2 performs mode composition with a mode composition coefficient, and outputs a drive command signal (current, voltage) to electric magnets of the radial magnetic bearings 1, 2.SELECTED DRAWING: Figure 14

The embodiment of the invention discloses a dialect speech synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a text feature corresponding to a target dialect speech, the text feature comprising a phoneme feature vector and / or a character feature vector corresponding to a text and a position coding vector of a text sequence; obtaining a Mel frequency spectrum of the target dialect speech according to the text features, and inputting the Mel frequency spectrum into a pre-trained optimization model to obtain an optimized Mel frequency spectrum; and inputting the optimized Mel spectrum into a pre-trained speech synthesis model to obtain a speech waveform of the target dialect speech. By implementing the embodiment of the invention, the quality of dialect speech synthesis can be improved under the condition that low-resource dialect corpora are scarce.

The invention discloses a 3D digital human facial expression synthesis method based on audio driving and a visualization system. The method comprises the following steps: acquiring a data set; inputting the original audio in the data set into an audio encoder in the model, and extracting audio features; inputting the audio features into a KAN-based decoder in the model to generate 3D digital human facial expression actions; inputting 3D digital human facial expression actions into a graph encoder in the model, and extracting lip features; collaborative optimization and model training of multi-modal features are realized by constructing a joint loss function fusing audio features, 3D facial expression parameters and lip features; and inputting an original audio to be tested into the trained model, and generating corresponding 3D digital human facial expression actions through the audio encoder and the KAN decoder in sequence. According to the invention, the generated 3D digital human facial expression action is more vivid and closer to the real character expression.

According to the feature fusion speech synthesis method and device based on the VITS, the expressive force and naturalness of speech can be improved, high-quality speech which is more natural, smoother and rich in expressive force is generated, and the expression is particularly outstanding in speech synthesis of multiple languages, multiple emotion expressions and complex contexts. The method comprises the following steps: (1) generating rhythm features matched with input text information according to the input text information; (2) establishing a deep learning model based on Transform; (3) comprehensively processing the rhythm features and the text features by adopting a multi-modal feature fusion method; (4) inputting the comprehensive feature vector into a speech synthesis network based on a variational auto-encoder for processing, wherein the speech synthesis network is trained by adopting a generative adversarial network and a training method based on maximum likelihood; and (5) adopting a neural network model of a random time length predictor, and performing input data transformation through a multi-layer normalized flow structure.

The invention provides a multimedia file synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: decoding a first multimedia file to obtain video and audio frame data, constructing audio waveform data, and analyzing statistical parameters and a specific time interval average value of the audio waveform data; determining a control parameter to regulate and control the special effect output of the video frame, synthesizing target video data, combining the target frame and the audio frame to generate a second multimedia file, and dynamically displaying the special effect based on the audio content when the file is played; the video special effect can be controlled through audio data analysis, and the function of dynamically displaying rich visual effects according to the audio content when the multimedia file is played is achieved.

The present disclosure provides a digital human synthesis method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The method includes: obtaining a target image, a driving video and a target audio; extracting three-dimensional face data and three-dimensional implicit key points in the target image, as well as the audio features of the target audio; predicting the three-dimensional face data mapped by the audio features based on the three-dimensional face data and the audio features; obtaining the head posture data of the character in the driving video, and predicting the motion optical flow of the target image based on the head posture data, the three-dimensional implicit key points and the three-dimensional face data mapped by the audio features; synthesizing the video frame image of the digital human based on the motion optical flow to obtain a video of the digital human broadcasting the target audio. The present disclosure ensures that the detailed description of the digital human's face and the head swing are continuous and natural, and improves the effect and stability of the digital human broadcasting video.

Disclosed in the present invention are a method and apparatus for synthesizing spatially cross-correlated multi-point ground motions fitting a response spectrum. The method comprises: using an influence matrix method to obtain ground motion acceleration time histories fitting a target reaction spectrum with high precision, and calculating a Fourier amplitude spectrum and a phase spectrum thereof; and considering a calculation relationship between a power spectral density function and the Fourier amplitude spectrum, taking the power spectral density function of the obtained time histories as an initial value, using a random vibration theory and Cholesky decomposition to obtain spatially cross-correlated multi-point ground motion time histories, further taking into account a phase spectrum of the ground motion time histories fitting the target reaction spectrum with high precision, and performing correction and iteration to obtain spatially cross-correlated multi-point ground motions fitting a response spectrum. The present invention overcomes the shortcomings in current multi-point ground motion time history synthesis techniques, such as the difficulty in ensuring good fitting to a target response spectrum and the non-stationarity of synthesized ground motions, helping to improve the reliability of dynamic response analysis results for large-scale engineering structures under the action of multi-point ground motions.

The invention relates to the technical field of intelligent decision making, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a speech synthesis method, device, equipment and medium based on an optimization strategy algorithm, comprising: extracting a time sequence processing network unit and a data sampling scheduling unit in a speech synthesis model; mapping a denoising function in the time sequence processing network unit into a multi-step Markov decision function, and converting an ordinary differential equation in the data sampling scheduling unit into a multi-source stochastic differential equation; sampling multiple groups of independent audio tracks corresponding to the input text based on a multi-source stochastic differential equation; calculating a strategy gradient modulation factor by using an optimization strategy algorithm and a multi-step Markov decision function; optimizing strategy parameters of the speech synthesis model according to the strategy gradient modulation factor to obtain an optimized speech synthesis model; and obtaining a to-be-converted text, and synthesizing voice corresponding to the to-be-converted text by using the optimized voice synthesis model. And the speech synthesis accuracy is improved.

The invention provides a defect synthesis method of an SMT (Surface Mount Technology) element based on cGAN network driving, which comprises the following steps: acquiring an input condition, constructing an adversarial loss function, an L1 reconstruction loss function, a circulation loss function and a physical loss function, and carrying out weighted summation on the four loss functions to obtain a total loss function. Training the cGAN network according to the total loss function to obtain an SMT element defect synthesis model; generating a virtual defect sample by adopting an SMT element defect synthesis model; wherein the virtual defect sample is used for reducing the false detection rate of the SMT defect detection model. The virtual defect sample generated by the SMT element defect synthesis model has visual authenticity, and the defect form in the virtual defect sample and the real welding process have physical consistency. The virtual defect sample generated by the SMT element defect synthesis model is combined with the real defect sample, so that the trained SMT defect detection model is suitable for SMT defect detection under a small sample condition, and the SMT defect detection precision under a real processing scene is improved.

An audio synthesis method, an audio synthesis model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which relate to artificial intelligence technology. The method includes: invoking an audio synthesis model based on language information and preset style information of a target text to perform following processing, the audio synthesis model including a prior encoder and a waveform decoder: generating audio features corresponding to the target text based on the language information and the preset style information by using the prior encoder; performing normalizing flow processing on the audio features by using the prior encoder, to obtain a hidden variable of the target text; and performing waveform decoding on the hidden variable of the target text by using the waveform decoder, to obtain a synthetic waveform conforming to an audio style described in the preset style information and corresponding to the target text.

The invention belongs to the technical field of text-to-speech conversion, and particularly relates to a solution for TTS (text-to-speech) in a high-concurrency scene through tone cloning. According to the method, the text can be generated and returned to the client by segmenting and parallelizing the text and reducing the delay of the first segment, compared with a traditional serial whole segment synthesis mode, the waiting time of a user is greatly shortened, the audio is returned after being generated, the waiting time of the user can be effectively shortened under high concurrency, the network bandwidth and the delay pressure are reduced, and the user experience is improved. The real-time performance is guaranteed, reconnection disasters are avoided, underlying model services can be fully utilized, the influence of the serial processing characteristic of a CosyVoice2 model is avoided, a client can immediately start to play a first segment of audio, even if subsequent segments are still synthesized, the continuous playing experience of previous content is not influenced, and when the concurrency is increased, the continuous playing experience of the previous content is not influenced. The model gateway can dynamically disperse requests to a plurality of GPU instances, and long queue waiting after a single-card video memory is fully occupied is avoided.

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: selecting a target speech synthesis model from a plurality of speech synthesis models based on the speech synthesis preference of a user, and carrying out the speech synthesis through employing the target speech synthesis model based on a to-be-synthesized text, and obtaining a synthesized speech meeting the speech synthesis preference; each speech synthesis model is obtained based on preference data sets corresponding to a plurality of preferences and preference weight configuration training, and the preference weight configurations used during training of different speech synthesis models are different, so that the defect that in a traditional scheme, a human preference alignment method for the speech synthesis models can only align single human preferences is overcome; according to the invention, the defects that the speech synthesis model cannot meet diversified human preference requirements in the prior art are overcome, the speech synthesis model is trained in a multi-target direct preference optimization mode, the training process is simpler and more efficient, human preference alignment can be carried out from multiple dimensions, and the diversified human preference requirements are met by giving preference weights.

The invention aims to serve the national strategic emerging industry, and discloses a deterministic material synthesis method and a deterministic material synthesis system for pre-nucleation chiral intervention (PNChI for short), which are suitable for one-dimensional or quasi-one-dimensional material systems (such as carbon nanotubes, biomacromolecules and the like) with formalized chiral tags. According to the method, before irreversible closed nucleation of a precursor is completed, deterministic authorization of a target chiral index (n, m) is realized in a controllable time window through at least one engineering intervention mechanism (such as physical field regulation, catalytic interface design, information coding guidance and the like) based on distinguishable physical properties, and consistency is kept in subsequent growth. The system comprises a programmable intervention unit and an intelligent sequence control module, and does not depend on a separation or screening step after nucleation. The single chiral index abundance in the obtained material is greater than or equal to 95%, preferably greater than or equal to 98%. The statistical limitation of traditional chiral control is broken through, a general synthesis normal form from'instruction input 'to'structure output' is constructed, and the method is suitable for high-consistency application scenes such as semiconductors, quantum devices, catalysts and biological recognition.

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and relates to the technical field of speech, and the method comprises the steps: inputting an obtained to-be-synthesized text and an emotion attribute into a speech synthesis model, and obtaining a target speech outputted by the speech synthesis model; wherein the speech synthesis model is obtained through training based on a first sample text corresponding to the first sample speech and a first sample emotion feature corresponding to the first sample speech, and the first sample emotion feature is obtained after the first sample speech is input into the emotion coding model; the emotion coding model is obtained by training based on the target code and the minimum mutual information loss of the second sample emotion feature. According to the invention, the emotion coding model can be obtained based on the minimum mutual information loss training, so that the emotion features output by the emotion coding model do not include unrelated information such as timbre and text content, the speech synthesis model can realize fine-grained emotion control, and the control force of speech synthesis is improved.

The invention discloses a low-delay speech synthesis method and device, a medium and equipment, and belongs to the technical field of speech processing, and the method comprises the steps: S10, carrying out the normalization processing of an input text, and obtaining a normalized text; s20, performing word segmentation processing, part-of-speech tagging processing and rhythm information analysis on the standardized text to obtain a language structured text; s30, performing feature extraction on the language structured text by using a deep learning algorithm to obtain a text feature vector; and S40, converting the text feature vector into a voice signal by using a non-autoregressive voice synthesis model based on a converter. According to the invention, the delay time of speech synthesis is reduced, and the naturalness and expressive force of speech are improved.

The invention provides a personalized speech synthesis method, device and equipment based on scene perception and natural language description, relates to the technical field of speech synthesis, and aims to generate speech which is personalized and expressive and conforms to the context. The method comprises the steps that semantic analysis is conducted on a first text to obtain a first semantic feature, and the first text comprises text content and scene description; scene classification generation is carried out according to the first semantic feature, a scene vector is obtained, and the scene vector represents a scene category; generating a scene sound feature according to the scene vector; semantic analysis is carried out on a second text to obtain a second semantic feature, and the second text at least comprises natural language description of the voice style; performing emotion category prediction and style feature extraction according to the second semantic features to obtain style control parameters; and according to the scene voice features and the style control parameters, adjusting synthetic voice parameters of the text content to obtain synthetic voice.

The invention provides a speech video synthesis method and system, and the method comprises the steps: inputting speech audio into a pre-trained audio-motion encoder, extracting a speech feature sequence from the speech audio, and generating a motion sequence of a face according to the speech feature sequence; inputting a single two-dimensional face picture shot from a first visual angle into a pre-trained picture encoder, and extracting three-dimensional identity features of an object contained in the face picture; inputting the three-dimensional identity features and the emotion labels into a pre-trained emotion mapping layer, and fusing to obtain emotion-containing three-dimensional identity features; the action sequence and the emotion-containing three-dimensional identity features are input into a video generation network based on a neural radiation field, a speech video of the object matched with the speech audio at a required second view angle is synthesized through the neural radiation field and camera parameters, and the second view angle can be adjusted within a preset range through the camera parameters.

The invention relates to the technical field of speech synthesis, in particular to a speech synthesis method and device, electronic equipment and a readable storage medium, and the method comprises the steps: obtaining a target text of a to-be-synthesized speech and a random sampling noise; inputting the target text into a text feature extraction module of a trained target speech synthesis model, and determining text features of the target text; the target speech synthesis model further comprises a text feature coding module and a speech generation module; inputting the text features and the random sampling noise into a text feature coding module, and determining potential variables of audio features corresponding to the text features; and inputting the potential variable into a voice generation module, generating an audio signal corresponding to the potential variable, and determining the audio signal as a target synthetic voice matched with the target text. Thus, through the trained speech synthesis model, the potential variable of the audio feature corresponding to the text feature of the target text can be directly converted into the target synthesis speech matched with the target text, and the synthesis efficiency of speech synthesis is improved.

The invention relates to a face synthesis method based on multi-modal information interaction, and relates to the technical field of video processing. The multi-modal information interaction-based face synthesis method comprises the steps of receiving audio clips and face images corresponding to the audio clips respectively; based on the frequency information of the audio clips, extracting audio time sequence features corresponding to the audio clips, and extracting audio semantic features from the audio clips through a machine learning algorithm; fusing the audio time sequence features and the audio semantic features through a bidirectional cross attention algorithm to obtain phonetic sequence semantic features; extracting facial features corresponding to the face image through a deep learning algorithm, and aligning the facial features with the audio semantic features to obtain facial tone semantic features; and fusing the phonetic sequence semantic feature and the facial tone semantic feature to obtain a joint feature, decoding and reconstructing the joint feature, and converting the joint feature into a target voice video. By implementing the method provided by the invention, the synchronism between the video frame and the audio can be improved.

A speech synthesis method, comprising: acquiring a target text to be synthesized into a speech and a target emotion category (S101); on the basis of a source speaker category and the target emotion category, processing the target text into source speech data having a source speaker timbre and the target emotion category (S102); determining from the source speech data phoneme feature data corresponding to the target text (S103); and, on the basis of the source speech data and a target speaker category, processing the phoneme feature data into a speech signal having a target speaker timbre and the target emotion category (S104). The method allows synthesized speeches to keep target emotion categories and target speaker timbres.