Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

1138 results about "Synthesis methods" patented technology

Synthesizers use various methods to generate electronic signals (sounds). Among the most popular waveform synthesis techniques are subtractive synthesis, additive synthesis, wavetable synthesis, frequency modulation synthesis, phase distortion synthesis, physical modeling synthesis and sample-based synthesis.

Speech synthesis method and device, computer equipment and storage medium

PendingCN119068863APreserve semantic accuracyFlexible adaptation to individual needsBiological modelsSpeech synthesisFeature vectorSynthesis methods
The invention discloses a speech synthesis method and device, computer equipment and a storage medium. The embodiment of the invention belongs to the field of artificial intelligence and voice generation, and relates to a voice synthesis method, which comprises the following steps: obtaining a to-be-processed text and to-be-processed voice acoustic features, carrying out vector analysis on the to-be-processed text through a text encoder to obtain a text feature vector, performing style analysis on the to-be-processed voice acoustic features through a style encoder to obtain a style vector, inputting the text feature vector and the style vector into a style diffusion model to perform style diffusion processing to obtain a style hidden variable, and importing the style hidden variable into a voice decoder to obtain the to-be-processed voice acoustic features. And sending the style hidden variable to the voice decoder, so that the voice decoder synthesizes the style hidden variable and the to-be-processed text to obtain a target style voice. The invention further provides a speech synthesis device, computer equipment and a storage medium. The speech synthesis method and device can adapt to personalized requirements of users and constantly changing application scenes, do not need to depend on a large amount of annotation data, and improve the flexibility and expansibility of a speech synthesis system.
Owner:PING AN TECH (SHENZHEN) CO LTD

Sparse planar array synthesis method

The present invention relates to the field of array antenna synthesis, and in particular to a sparse planar array synthesis method. In the present invention, in order to balance the proportion of a development and exploration process, a nonlinear function is used to reconstruct an arithmetic optimization accelerator in an arithmetic optimization algorithm; the top three optimal individuals are used to replace the current optimal individual to perform exploration and development and an elite mutation strategy is introduced, so as to enhance the capability of the algorithm to escape a local optimum, and improve the convergence precision of the algorithm; and an adaptive matrix mapping rule is provided, the current array element distribution is determined, and if the current array element distribution does not meet the minimum array element spacing constraint, the current array element distribution is adjusted by means of an adjustment strategy, thereby guaranteeing the degree of freedom of array elements while avoiding an unfeasible solution.
Owner:HARBIN ENG UNIV

Speech synthesis method and device, electronic equipment and storage medium

The invention provides a speech synthesis method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a to-be-synthesized text and a tone reference audio; inputting the to-be-synthesized text into a pre-training decoding model to enable the pre-training decoding model to determine voice content and voice style corresponding to the to-be-synthesized text, and obtaining semantic features corresponding to the to-be-synthesized text; wherein the pre-training decoding model is trained by using a training text and a corresponding training semantic feature, the training semantic feature is generated by using a training voice audio corresponding to the training text, and the training voice audio has a preset voice style; inputting the semantic features, the to-be-synthesized text and the timbre reference audio into a pre-trained speech synthesis model for speech synthesis to obtain a synthesized speech audio; according to the invention, speech synthesis can be carried out together with the text to be synthesized and the timbre reference audio by using the semantic features containing the speech content and the speech style, so that the timbre style decoupling is realized.
Owner:SHANGHAI SOULGATE TECH CO LTD

Voice synthesis method and device based on gated attention mechanism, equipment and medium

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech synthesis method based on a gated attention mechanism, and the method comprises the steps: obtaining text data and speech data, converting the text data into a text symbol sequence through a text encoder, speech features of the speech data are extracted through a speech encoder trained based on a self-supervised learning model, and the speech features are quantized into a discrete speech symbol sequence; then, respectively organizing the text symbol sequence and the voice symbol sequence into a text sequence and a voice sequence; the corresponding relation between the text symbols and the voice symbols is established through preliminary alignment processing, the matching degree between the text symbols and the voice symbols is dynamically adjusted through a gating attention mechanism, and finally a final voice signal is generated through a decoder. According to the invention, through the voice encoder trained by the self-supervised learning model, the extraction capability of voice features is effectively improved, and effective feature representation can still be learned from unlabeled data especially in a scene lacking a large amount of labeled data.
Owner:PING AN TECH (SHENZHEN) CO LTD

Cross-language voice migration synthesis method and device, equipment and medium

The invention relates to the technical field of voice processing, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a cross-language voice migration synthesis method, device, equipment and medium. And training an acoustic model by adopting a hierarchical self-adaptive fine tuning strategy, fusing the phoneme sequence for reasoning and the tone mark to generate a representation sequence, and finally synthesizing a target language speech signal. According to the method, a sharing and separation parallel cross-language modeling structure is constructed, the accuracy of phoneme and tone modeling in a low-resource language is effectively improved, the target language acoustic model has higher generalization ability and migration efficiency in combination with a multi-stage self-adaptive fine tuning and training data enhancement strategy, and the method is suitable for being applied to the field of multi-language modeling. And finally, synchronous improvement of the voice naturalness and the tone fidelity is realized.
Owner:PING AN TECH (SHENZHEN) CO LTD

Speech synthesis method and device, electronic equipment, storage medium and program product

The invention relates to a speech synthesis method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a source language dubbing audio corresponding to a source language subtitle text of a source video and a target language subtitle text translated by the source language subtitle text; extracting audio emotion features from the source language dubbing audio by using an emotion extractor, wherein the audio emotion features represent emotions expressed by the source language dubbing audio; converting the target language subtitle text into a phoneme sequence, and encoding the phoneme sequence by using a text encoder to obtain text encoding features; fusing the audio emotion features with the text coding features to obtain emotion text features; and generating a target language audio based on the emotion text features by using a decoder, the target language audio being used as a dubbing audio of the source video in the target language. Therefore, the high-quality target language audio with the source dubbing audio emotion can be automatically and efficiently generated, the cost is low, and the efficiency is high.
Owner:YOUKU CULTURE TECH (BEIJING) CO LTD

Multi-agent-based material performance prediction and synthesis method and system

The invention relates to a multi-agent-based material performance prediction and synthesis system, and the system comprises a multi-agent data enhancement module which is configured to be used for firstly disassembling a complex problem into a plurality of subtasks, and then constructing a fine tuning data set comprising Sub-CoQ question and answer pairs by starting multi-source parallel retrieval; the multi-expert debate module is configured to be used for simulating decision conflicts of different roles in material engineering and generating a direct preference optimization DPO data set through debate; the training and verification module is configured to be used for training and verifying a large model MatMind in the field of materials by utilizing supervised fine tuning SFT and reinforcement learning RLHF based on the fine tuning data set and the DPO data set; and the material performance prediction and synthesis module is configured to be used for realizing intelligent recommendation of a material performance prediction and synthesis process by importing input parameters into the large model MatMind.
Owner:SHANGHAI INST OF CERAMIC CHEM & TECH CHINESE ACAD OF SCI

Speech synthesis method and device, computer equipment and storage medium

The invention belongs to the field of artificial intelligence, medical treatment and finance, and relates to a speech synthesis method, which comprises the following steps: performing phoneme conversion and context embedding on a text data set, obtaining a phoneme sequence and a text context vector, extracting reference speech features of a reference audio sample, inputting a phoneme-level potential diffusion model, and generating phoneme-level prior potential features; inputting the phoneme-level prior potential features into a phoneme-level variational self-encoding model to obtain reconstructed speech features; extracting a target voice feature of the target audio sample; performing alignment and up-sampling on the target voice feature and a phoneme sequence to obtain a target audio frame-level feature; and training based on the reconstructed speech features and the target audio frame-level features to obtain a conditional control speech synthesis model, and synthesizing the to-be-converted text speech. The invention further provides a speech synthesis device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted text can be stored in a block chain. The speaker fidelity and the text understandability are improved.
Owner:PING AN TECH (SHENZHEN) CO LTD

High-expressive-force singing sound synthesis model training method, synthesis method and device

The invention provides a training method, a synthesis method and a device for a high-expressive singing sound synthesis model. The method comprises the following steps: acquiring a lyric text and a music score sequence obtained by predicting real singing sound; converting the lyric text into phoneme information, extracting physical information of the singing sound from the singing sound audio data, and analyzing the physical information to obtain intensity information; the phoneme information and attribute information of the music score sequence are input into a text encoder to generate a text embedding sequence, the duration of each word is determined by a duration predictor, and the text embedding sequence and the duration are input into a length constraint device to generate voice representation; inputting the speech representation into a fundamental frequency decoder and a Mel decoder to generate a fundamental frequency and a Mel spectrum; and adding the fundamental frequency, the Mel spectrum and the voice representation to obtain a mixed embedded vector, inputting the mixed embedded vector into a Transformer and a continuous voice unit representation module to generate a continuous voice unit vector, and synthesizing a predicted singing sound by a vocoder. According to the method provided by the invention, fine-grained modeling is carried out on each attribute, coupling is avoided, and high-quality singing sound is synthesized.
Owner:SHANGHAI AITEOU DIGITAL ENTERTAINMENT TECHNOLOGY CO LTD

Speech synthesis method, device, equipment and computer medium

The invention provides a speech synthesis method, device and equipment and a computer readable medium, and the method comprises the steps: obtaining input data, the input data comprises a reference audio and a target synthesis text, and then determining a target tone feature and a reference speech content feature sequence according to the reference audio, reference voice content features in the reference voice content feature sequence are used for representing at least one of audio content, intonation, rhythm and speaking habits of the reference voice, and inputting the reference voice content feature sequence and the target synthetic text into a pre-trained voice content feature generation model, according to the method, the target voice content feature sequence is generated, and then the target synthetic audio is generated according to the target timbre feature and the target voice content feature sequence, so that the target synthetic text can be converted into the target synthetic audio conforming to the timbre, the intonation, the rhythm, the speaking habit and the like of the reference audio, and the authenticity of voice synthesis is improved.
Owner:SHANGHAI HODE INFORMATION TECH CO LTD

Speech synthesis method and device, equipment and medium

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device, equipment and a medium, and the method comprises the steps: obtaining a to-be-synthesized text and a reference audio; dividing the to-be-synthesized text into a plurality of text segments by adopting a sliding window algorithm; generating an audio waveform corresponding to each text segment based on the reference audio and a pre-trained speech synthesis model; and performing speech synthesis on the audio waveform corresponding to each text segment to obtain a target speech. Through the mode, the input text is segmented by adopting the sliding window algorithm to form a plurality of smaller and easier-to-manage text segments, and the plurality of text segments and the parameter audio are input into the speech synthesis model, so that the speech synthesis model generates the audio sentence by sentence based on the shorter text segments; therefore, the model can focus on accurate voice and intonation details of each segment, the accuracy and natural fluency of audio output are ensured, the degradation problem caused by the fact that the text length exceeds a typical length parameter is solved, and the stability of voice generation is improved.
Owner:PING AN TECH (SHENZHEN) CO LTD

Sound duplicating and low-delay streaming speech synthesis method and system based on ultra-short sample

The invention provides a voice replication and low-delay streaming voice synthesis method and system based on an ultra-short sample, relates to the technical field of artificial intelligence, and is suitable for intelligent interaction, outbound service and multi-mode communication scenes. Deep personalized customization of intelligent voice interaction and real-time generation of ultra-low delay are realized, and bidirectional streaming interaction of a system level is supported, so that the fluency and response speed of dialogues are improved; an ultra-short sample sound duplicating module specially designed for processing ultra-short audio samples and a speech synthesis engine with ultra-low delay and bidirectional streaming output capability are integrated, and the capabilities of the ultra-short sample sound duplicating module and the speech synthesis engine are applied to a real-time and interactive intelligent speech interaction process to form a complete and efficient solution. The industrial pain point is directly solved in a targeted manner, and the method has important commercial application value.
Owner:GUANGDONG CHAOTENG INFORMATION TECHNOLOGY CO LTD

Speech synthesis model training method and device, speech synthesis method and device and storage medium

The invention provides a speech synthesis model training method, a speech synthesis method, a speech synthesis device and a storage medium, and the speech synthesis model training method comprises the steps: obtaining the sample text data of a plurality of speakers and the acoustic features of the sample audio data of the plurality of speakers according to the sample audio data of the plurality of speakers, feature extraction is carried out on the sample text data of each speaker to obtain sample text features of each speaker, a speech synthesis sample of each speaker is constructed according to the sample text features of each speaker and acoustic features of the sample audio data of each speaker, and speech synthesis is carried out according to the speech synthesis samples of multiple speakers. And respectively carrying out model training to obtain speech synthesis models of the plurality of speakers. Automatic training of the speech synthesis model is carried out in a one-button mode, and the iteration efficiency of the speech synthesis technology is improved.
Owner:NETEASE (HANGZHOU) NETWORK CO LTD

Phoneme time axis driven high-definition video mouth shape automatic synthesis method

The invention discloses a high-definition video mouth shape automatic synthesis method driven by a phoneme time axis, and relates to the technical field of electrical digital data processing, and the method comprises the steps: 1, obtaining an input text or voice, employing a fixed phoneme set and a deterministic pronunciation rule to generate a phoneme sequence and a corresponding starting and ending time point, and building the phoneme time axis; outputting the phoneme time axis, the frame index mapping table and the target frame set; 2, on the basis of a fixed and limited mouth shape dictionary and a pronunciation part rule, three complementary derivation chains are established at the same time, and a mouth shape key frame sequence meeting phase continuity and boundary traceability is obtained; and step 3, extracting three-dimensional face feature points and a head posture track based on the original video, and outputting a video mouth shape synthesis result which is consistent with the resolution of the original video and is synchronous with a phoneme time axis according to the posture uniformization mouth shape sequence. According to the invention, the accuracy and reality of mouth shape video generation are obviously improved.
Owner:CLOUD ATTACK NETWORK TECH HEBEI CO LTD

Method and system for synthesizing any digital signal

The invention discloses an arbitrary digital signal synthesis method and system. The method comprises the following steps: analyzing a target parameter of a digital signal to be synthesized; generating a first multi-path baseband waveform by using a digital direct frequency synthesizer according to a modulation rate parameter in the target parameters, and reading pre-stored noise data to generate a second multi-path baseband waveform; selecting a plurality of appointed baseband waveforms, and modulating the plurality of baseband waveforms by using carrier frequency and bandwidth parameters in target parameters to obtain a plurality of instantaneous frequency curves; and integrating the multiple instantaneous frequency curves to obtain a phase stepping curve of each path, selecting a phase modulation curve according to a phase modulation type in the target parameters, and performing digital direct frequency synthesis on the phase stepping curve and the phase modulation curve of each path to generate multiple parallel digital synthesis signals. According to the invention, the synthesis process of the digital signals of different modulation types is uniformly decomposed, so that the coupling degree of the module and the complexity of the system are greatly reduced, and meanwhile, the resource consumption is reduced.
Owner:HUNAN ECONOVEL TECH CO LTD

Circuit synthesis method and apparatus, electronic device and readable storage medium

Provided in the embodiments of the present invention are a circuit synthesis method and apparatus, an electronic device and a readable storage medium, the method comprising: analyzing an assertion statement to be synthesized, and identifying a keyword and a verification mode that correspond to the assertion statement; instantiating the keyword in the assertion statement into a basic circuit module; on the basis of the verification mode, determining a connection mode corresponding to the basic circuit module; and, on the basis of the connection mode, connecting the basic circuit module so as to obtain an assertion circuit corresponding to the assertion statement.
Owner:BEIJING INSTITUTE OF OPEN SOURCE CHIP

Method for synthesizing 2h / 1t' TMD heterophase junctions and a device based on the same

Provided is a method for synthesizing 2H / 1T′ transition metal dichalcogenide (TMD) heterophase junctions. Synthesis of vertical 2H / 1T′ MoS2 heterophase junction is achieved by growing 2H MoS2 monolayers on a mica substrate under pure Ar carrier gas, and then growing 1T′ MoS2 nanoribbons on top of the 2H MoS2 monolayers by changing the carrier gas from pure Ar to a mixture of Ar and H2 gases. The integrated devices based on 2H / 1T′ MoS2 heterophase structures display typical rectifying behavior with a current rectifying ratio of over 102 and achieve a responsivity of 1.07 A / W at 532 nm with an ultra-low dark current of lower than 10−11 A. These superior features make 2H / 1T′ MoS2 heterophase structure a promising candidate for future rectifiers as well as photodetectors. More importantly, the provided synthesis method can pave the way toward tailoring the phase in MoS2 to promote the performance of electronic devices.
Owner:CITY UNIVERSITY OF HONG KONG

Hierarchical coding and decoding speech synthesis method, device, equipment and medium

The invention relates to the technical field of speech synthesis, can be applied to business system platforms such as financial science and technology, medical health and the like, and discloses a layered coding and decoding speech synthesis method, device, equipment and medium, and the method comprises the following steps: carrying out semantic feature extraction on a pre-acquired original text to obtain text semantic features; performing prosodic feature extraction on the original text to obtain a text prosodic feature; performing feature fusion compression on the text semantic features and the text rhythm features to obtain fusion compression features; performing low-frequency decoding on the fusion compression features to obtain low-frequency decoding features; performing high-frequency decoding on the fusion compression feature to obtain a high-frequency decoding feature; and combining the low-frequency decoding features and the high-frequency decoding features, and converting the combined features into target synthetic speech. Through high and low frequency decoding, richer and more accurate voice details are restored, so that the synthesized voice is listened more naturally and smoothly, and the credibility and fluency of the voice are improved.
Owner:PING AN TECH (SHENZHEN) CO LTD

Speech synthesis model training method, speech synthesis method, electronic device, and storage medium

The present disclosure provides a speech synthesis model training method. The method can comprise: acquiring initial training data; selecting a plurality of first text segments, second text segments, and third text segments from coherent text, and acquiring first audio segments, second audio segments, and third audio segments from coherent audio; detecting whether the plurality of audio segments and text segments meet a stitching condition; on the basis of the text sequence, stitching, by alternatively arranging text and audio, the plurality of text segments and audio segments meeting the stitching condition to generate combined training data, so as to generate a combined training data set; on the basis of the combined training data set, training the initial speech generation model to obtain a trained speech synthesis model, wherein during the training, prosodic, tonal, and / or emotional features from the second text segments and / or the third text segments are extracted. Further disclosed is a speech synthesis method. The present disclosure allows for use of the context of the text and audio to realize text comprehension to capture prosody, tonality, and emotion in the context.
Owner:SHANGHAI XIYU JIZHI TECH CO LTD

Speech synthesis method and device, vehicle and storage medium

The embodiment of the invention provides a voice synthesis method and device, a vehicle and a storage medium, and the method comprises the steps: obtaining multi-modal data which comprises a voice signal, a user instruction, a historical interaction log and bus data; determining model input parameters according to the multi-modal data; according to the model input parameters, the static corpus and a preset large model, generating a personalized script, the preset large model being used for dynamically generating the personalized script in combination with the model input parameters and the static corpus; and synthesizing the broadcast audio according to the personalized script. According to the invention, the technical problem that the voice synthesis technology in the related technology cannot adapt to the personalized demands of the user is solved.
Owner:GUANGZHOU AUTOMOBILE GROUP CO LTD

Speech synthesis method, device, equipment, medium and product

The invention provides a speech synthesis method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is obtained by performing multi-stage prediction task training according to a phoneme sequence and a discrete acoustic feature sequence corresponding to the training text, and a multi-stage prediction task comprises a discrete acoustic feature sequence prediction task and a Mel spectrum feature prediction task. The quality of synthesized speech can be improved.
Owner:IFLYTEK CO LTD

Control device and control method for magnetic bearing

To provide a control device for a magnetic bearing that enables mode separation of a natural vibration mode with high precision and can design a stable controller with the degree of the controller made small.SOLUTION: With respect to a model for a rotary body controlled by radial magnetic bearings 1, 2, a status space model is structured by excluding a controller which controls a plant from a mode composition model having parameters of a mass matrix, a rigidity matrix, and an attenuation matrix identified and adjusted by a parameter estimation part, and a mode separation part 61 on the input side of controllers 1, 2 inputs a measurement signal of a displacement sensor from the radial magnetic bearings 1, 2 to perform mode separation with a mode separation coefficient. A mode composition part 62 on the output side of the controllers 1, 2 performs mode composition with a mode composition coefficient, and outputs a drive command signal (current, voltage) to electric magnets of the radial magnetic bearings 1, 2.SELECTED DRAWING: Figure 14
Owner:MEIDENSHA CORP

Dialect speech synthesis method and device, electronic equipment and storage medium

The embodiment of the invention discloses a dialect speech synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a text feature corresponding to a target dialect speech, the text feature comprising a phoneme feature vector and / or a character feature vector corresponding to a text and a position coding vector of a text sequence; obtaining a Mel frequency spectrum of the target dialect speech according to the text features, and inputting the Mel frequency spectrum into a pre-trained optimization model to obtain an optimized Mel frequency spectrum; and inputting the optimized Mel spectrum into a pre-trained speech synthesis model to obtain a speech waveform of the target dialect speech. By implementing the embodiment of the invention, the quality of dialect speech synthesis can be improved under the condition that low-resource dialect corpora are scarce.
Owner:CHINA UNICOM (GUANGDONG) IND INTERNET CO LTD

Near / span strike-slip fault seismic oscillation synthesis method for bridge structure

The invention discloses a near / span strike-slip fault seismic oscillation synthesis method for a bridge structure. The method comprises the following steps: step 1, selecting a near fault pulse type seismic record matched with a target response spectrum according to a bridge site and a wave selection principle; 2, rotating and changing the seismic record to parallel and vertical fault directions, and decomposing the seismic record into a high-frequency component and a low-frequency component by using a filter; and step 3, further decomposing the low-frequency component based on continuous wavelet variation, simulating a pulse component by using a pulse function, and introducing a particle swarm optimization algorithm to determine a fitting parameter. And 4, superposing the residual component, the artificial fitting pulse and the high-frequency component to obtain broadband near-fault seismic oscillation. And 5, selecting a pulse waveform with a slip effect and a directivity effect to replace the fitting pulse, extracting a sub-high-frequency component in the residual component, and finally superposing the sub-high-frequency component and the high-frequency component to obtain cross-fault seismic motion, thereby realizing accurate simulation of strike-slip fault ground motion.
Owner:SOUTHEAST UNIV

Intelligent robot speech synthesis method based on deep learning

The invention relates to the technical field of intelligent speech synthesis, and discloses an intelligent robot speech synthesis method based on deep learning, which comprises the following steps: 1, text processing: receiving an input text, and carrying out word segmentation, part-of-speech tagging and phoneme tagging on the text to obtain a text embedding sequence for a subsequent sentiment analysis step; step 2, sentiment analysis: according to the generated text embedding sequence, performing sentiment analysis on the text by using a BERT-based sentiment classification model to obtain text sentiment embedding for multi-modal sentiment fusion; and step 3, receiving visual input, extracting visual emotion embedding through the convolutional neural network, and inputting the visual emotion embedding into multi-modal emotion fusion. Through the deep learning model, voice data can be learned, natural phonemes, intonation and rhythm characteristics in voice can be extracted, a natural and smooth voice synthesis effect is generated, the voice speed, the intonation and the tone quality are close to those of human voice, and the auditory experience of a user is enhanced.
Owner:BEIJING AEROSPACE FLYING EAGLE INTELLIGENT ROBOT TECH CO LTD

3D digital human facial expression synthesis method and visualization system based on audio driving

The invention discloses a 3D digital human facial expression synthesis method based on audio driving and a visualization system. The method comprises the following steps: acquiring a data set; inputting the original audio in the data set into an audio encoder in the model, and extracting audio features; inputting the audio features into a KAN-based decoder in the model to generate 3D digital human facial expression actions; inputting 3D digital human facial expression actions into a graph encoder in the model, and extracting lip features; collaborative optimization and model training of multi-modal features are realized by constructing a joint loss function fusing audio features, 3D facial expression parameters and lip features; and inputting an original audio to be tested into the trained model, and generating corresponding 3D digital human facial expression actions through the audio encoder and the KAN decoder in sequence. According to the invention, the generated 3D digital human facial expression action is more vivid and closer to the real character expression.
Owner:SOUTH CHINA UNIV OF TECH

Feature fusion speech synthesis method and device based on VITS

According to the feature fusion speech synthesis method and device based on the VITS, the expressive force and naturalness of speech can be improved, high-quality speech which is more natural, smoother and rich in expressive force is generated, and the expression is particularly outstanding in speech synthesis of multiple languages, multiple emotion expressions and complex contexts. The method comprises the following steps: (1) generating rhythm features matched with input text information according to the input text information; (2) establishing a deep learning model based on Transform; (3) comprehensively processing the rhythm features and the text features by adopting a multi-modal feature fusion method; (4) inputting the comprehensive feature vector into a speech synthesis network based on a variational auto-encoder for processing, wherein the speech synthesis network is trained by adopting a generative adversarial network and a training method based on maximum likelihood; and (5) adopting a neural network model of a random time length predictor, and performing input data transformation through a multi-layer normalized flow structure.
Owner:BEIJING INSTITUTE OF GRAPHIC COMMUNICATION

Multimedia file synthesis method and device, electronic equipment and storage medium

The invention provides a multimedia file synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: decoding a first multimedia file to obtain video and audio frame data, constructing audio waveform data, and analyzing statistical parameters and a specific time interval average value of the audio waveform data; determining a control parameter to regulate and control the special effect output of the video frame, synthesizing target video data, combining the target frame and the audio frame to generate a second multimedia file, and dynamically displaying the special effect based on the audio content when the file is played; the video special effect can be controlled through audio data analysis, and the function of dynamically displaying rich visual effects according to the audio content when the multimedia file is played is achieved.
Owner:BEIJING TRICOLOR TECH

Digital human synthesis method, device, electronic equipment and storage medium

The present disclosure provides a digital human synthesis method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology. The method includes: obtaining a target image, a driving video and a target audio; extracting three-dimensional face data and three-dimensional implicit key points in the target image, as well as the audio features of the target audio; predicting the three-dimensional face data mapped by the audio features based on the three-dimensional face data and the audio features; obtaining the head posture data of the character in the driving video, and predicting the motion optical flow of the target image based on the head posture data, the three-dimensional implicit key points and the three-dimensional face data mapped by the audio features; synthesizing the video frame image of the digital human based on the motion optical flow to obtain a video of the digital human broadcasting the target audio. The present disclosure ensures that the detailed description of the digital human's face and the head swing are continuous and natural, and improves the effect and stability of the digital human broadcasting video.
Owner:XIAN DIANGAO NET TECH CO LTD

Method and apparatus for synthesizing spatially cross-correlated multi-point ground motions fitting response spectrum

Disclosed in the present invention are a method and apparatus for synthesizing spatially cross-correlated multi-point ground motions fitting a response spectrum. The method comprises: using an influence matrix method to obtain ground motion acceleration time histories fitting a target reaction spectrum with high precision, and calculating a Fourier amplitude spectrum and a phase spectrum thereof; and considering a calculation relationship between a power spectral density function and the Fourier amplitude spectrum, taking the power spectral density function of the obtained time histories as an initial value, using a random vibration theory and Cholesky decomposition to obtain spatially cross-correlated multi-point ground motion time histories, further taking into account a phase spectrum of the ground motion time histories fitting the target reaction spectrum with high precision, and performing correction and iteration to obtain spatially cross-correlated multi-point ground motions fitting a response spectrum. The present invention overcomes the shortcomings in current multi-point ground motion time history synthesis techniques, such as the difficulty in ensuring good fitting to a target response spectrum and the non-stationarity of synthesized ground motions, helping to improve the reliability of dynamic response analysis results for large-scale engineering structures under the action of multi-point ground motions.
Owner:JIANGNAN UNIV