240 results about "Speech translation" patented technology

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

The invention belongs to the technical field of speech translation, and particularly relates to a video speech automatic translation method based on a pre-training model, and the method comprises the following steps: 1, preprocessing video and audio data; step 2, voice recognition and language detection; step 3, machine translation and text post-processing; step 4, speech synthesis and audio mixing; step 5, synchronizing video processing and subtitles; step 6, quality control and multi-dimensional evaluation; 7, carrying out model iteration and data closed loop; and step 8, system deployment and engineering implementation. Through deep fusion of efficient transfer learning of the pre-training model and the multi-modal technology, a high-precision, low-cost and easy-to-expand video speech translation solution is constructed, the time and labor cost of globalized content production is greatly reduced, the cross-language communication efficiency is improved, immersive multi-language experience is provided, and the method is suitable for popularization and application. And a data-driven continuous optimization mechanism is established, so that the system performance is improved along with the increase of the use scale.

The invention relates to the technical field of artificial intelligence, and discloses a Bluetooth communication intelligent speech translation method and system based on multi-mode enhancement, and the method comprises the steps: collecting a multi-channel audio signal through a built-in multi-microphone array of a Bluetooth device, carrying out the dynamic direction self-adaptive beam forming of the multi-channel audio signal, and carrying out the self-adaptive beam forming of the multi-channel audio signal; extracting a Mel spectrogram feature of the direction enhancement signal, identifying lip regions of a plurality of candidate speakers in each frame of real-time speaking video captured by a camera, performing time sequence convolution on the lip regions to obtain a lip movement time sequence embedded vector, calculating a correlation score with the Mel spectrogram feature, separating the direction enhancement signal, and obtaining a lip movement time sequence embedded vector; and performing text transcription and conversion on the high-confidence separation voice to obtain a translation language text, and sending the synthesized target translation voice to a preset mobile terminal through the Bluetooth device to obtain a target translation result. According to the method, the real-time performance and accuracy of speech translation are improved in a multi-person scene, far-field speech, noise interference and accent difference.

The invention discloses a cross-modal attention, global memory and dynamic convolution-based speech translation model training method and device, and a speech translation method and device, and relates to the technical field of speech processing and machine translation. A speech translation model is designed to comprise a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model decoder, a global memory network, a dynamic convolution decoder and an output layer. The cross-modal attention adapter projects audio features and performs multi-head cross attention fusion with text embedding; the global memory network updates and enhances historical memory on the basis of a gating mechanism and a Transform Encoder; and the dynamic convolution decoder performs multi-scale convolution extraction on the decoded hidden representation and fuses with the memory, so that the translation quality is improved. According to the method, deep fusion of voice and text, continuous memory with contextual coherence and high-quality translation generation can be realized, the end-to-end voice translation performance is remarkably improved, and the actual requirements of real-time and high-quality end-to-end voice translation in a complex scene are met.

The invention relates to an intelligent real-time language synchronous translation system and a terminal thereof. The system comprises a voice segmentation module used for carrying out semantic analysis on target voice to be translated and segmenting the target voice according to obtained semantic information to obtain voice units; the scene recognition module is used for analyzing the voice unit to obtain scene information; the syntactic rhythm analysis module is used for converting the voice unit into a text, performing syntactic analysis, and analyzing the voice unit in combination with the information and the semantic information to obtain voice rhythm information and statement structure information; the semantic optimization module is used for correcting the semantic information based on the statement structure information, the scene information and the voice rhythm information to obtain optimized semantic information; and the speech translation module is used for translating the optimized semantic information based on the target language to obtain a target language translation. According to the system, through multi-module cooperation, voice semantics can be accurately mastered, different scenes are fitted, and the accuracy, smoothness and practicability of translation are effectively improved.

An automatic online voice translation system and method, comprising a virtual translator, a SIP gateway, and a storage, wherein the SIP gateway receives a call from a caller, when the virtual translator is not activated, the SIP gateway establishes a communication between the caller and the agent by streaming in real-time caller's voice directly to the agent and agent's voice directly to the caller, and when the virtual translator is activated, the SIP gateway creates audio files based on the received voice streams of both the caller and the agent, and sends the created audio files to the virtual translator for translating the caller's voice into a language understood by the agent, before transmitting the voice to the agent. In another embodiment, a contact center platform is used for applying features such as call recording, Interactive Voice Response (IVR) service, and conferencing service to the call.

The invention discloses a voice translation system based on a Bluetooth headset, which relates to the technical field of voice translation of Bluetooth headsets and comprises a voice acquisition module, a voice speed detection module, a rhythm evaluation module, a voice regulation and control module and a text verification module. The rhythm evaluation module is used for extracting rhythm characteristic information in a user voice signal collected by the Bluetooth earphone in the voice speed fluctuation time period, analyzing the rhythm characteristic information and evaluating the intensity of statement rhythm mutation in user expression in the voice speed fluctuation time period; and the voice regulation and control module is used for selecting and executing a corresponding voice processing strategy according to the evaluation result, and carrying out dynamic regulation and control in the execution process. According to the method, the problem that statement rhythm mutation caused by speech speed fluctuation cannot be dynamically regulated and controlled is solved, intelligent adaptation and whole-process response of a speech processing strategy are realized, structural optimization is performed on a translation result, and translation accuracy and semantic coherence are remarkably improved.

The invention provides a speech translation method, device, equipment and product, and is applied to the technical field of artificial intelligence. The speech translation method comprises the steps that speech to be translated and associated data of the speech to be translated are acquired, the associated data comprise associated images and / or associated videos, the associated images comprise image information of at least part of objects described by the speech to be translated, and the associated videos display lip actions when a speaking object expresses the speech to be translated; and based on the to-be-translated speech and the associated data, translating the to-be-translated speech through the multi-modal speech translation model to obtain a translation result of the to-be-translated speech. Therefore, in combination with the images and / or videos related to the voice, end-to-end voice translation is carried out through the multi-mode voice translation model, and the voice translation quality is effectively improved.

Embodiments of the present disclosure relate to a method and apparatus for speech translation, an electronic device, and a medium. The method includes obtaining an audio in a source language, where the audio includes a specific type of information. The method further includes obtaining prompt content related to a target language. In addition, the method further includes generating, based on the audio and the prompt content, a target-language text corresponding to the audio, where the target-language text includes a punctuation mark corresponding to the specific type of the information.

The invention provides a voice processing method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a target voice and a translation mode of the target voice; based on a voice encoder module and a translation mode in the voice processing model, encoding the target voice to obtain voice content features and acoustic features of the target voice; translating the target voice based on a large language model, a translation mode and voice content features in the voice processing model to obtain a target translation text of the target voice; and based on a voice decoder module, the translation mode and the acoustic features in the voice processing model, performing voice synthesis on the target translation text to obtain a target translation voice of the target voice. According to the invention, adaptive processing is carried out on voice coding, translation and voice decoding through a voice processing model of an integrated framework integrating voice translation and voice simultaneous transmission in combination with a translation mode, so that the deployment cost is better reduced, and the real-time performance and quality of voice processing are optimized.

The invention relates to a speech translation method, device and equipment of an end cloud translation system and a medium, and relates to the technical field of intelligent speech translations, the end cloud translation system comprises an audio receiving end and a cloud, the method is applied to the audio receiving end, and the method comprises the following steps: obtaining a first language audio of the audio receiving end; sending the first language audio to a cloud; and receiving and playing a second language audio, wherein the second language audio is translated by the cloud according to the first language audio and then is sent out. The method can be widely applied to cross-border conference scenes, and the cross-language communication efficiency and experience are improved.

The invention provides a voice, picture and subtitle alignment method and system for real-time voice translation synthesis, and relates to the technical field of voice processing. According to the method, millisecond alignment of translated voice and video is realized through fragment-level parallel processing, compared with a traditional scheme, the delay is greatly reduced, and the problem that translated voice pictures are asynchronous during live broadcast is solved; the method comprises the following steps: automatically generating real-time voice translation according to live broadcast content, and respectively processing to obtain a corresponding srt file and a translated m3u8 fragment; therefore, there is no need to manually translate and generate subtitles in advance, and there is no need to replace the audio of the original video in advance, thereby saving the manpower cost, and providing a more universal live broadcast stream subtitle scheme. Besides, when the languages are switched, the player can automatically select the starting time point, so that the method supports multiple playing modes such as online live broadcast and live broadcast playback.

The invention discloses a multi-modal language conversion method and a related product. The input data is coded through a multi-mode coding unit in the multi-mode decoding unit, vector representation of the input data is obtained, the vector representation is decoded through a plurality of decoding units in the multi-mode decoding unit, and target data obtained after language conversion is conducted on the input data is obtained and output. Wherein the plurality of decoding units are respectively obtained by carrying out association training on different basic decoding units, and each decoding unit is related to a multi-modal cross-language understanding function. By adopting the multi-mode coding unit and the multi-mode decoding unit, the problems of information loss and time delay caused by a chained processing mode in a traditional speech translation system are solved, and the balance between real-time performance and accuracy is improved.

A speech translation system may provide real-time or near real-time translation of speech uttered by a person or emitted from a media device. The speech translation system may include a device that may receive audio representing speech in a source language and output audio representing speech in a target language. The speech translation system may translate the speech in portions representing semantically cohesive speech segments such that the target speech reflects the semantic meaning of words, phrases, and / or clauses as used in the context of the source speech. The speech translation system may condense the speech segments prior to or during translation to reduce verbosity. The speech translation system may selectively translate some speakers and not others, and may determine voice characteristics of source speech and apply identifying characteristics to the target speech that allow a user to differentiate respective target speech from different speakers based on the identifying characteristics.

The invention provides an end-to-end Mongolian-Chinese speech translation method and system based on comparative learning, a storage medium and electronic equipment, and relates to the technical field of speech translation. The embodiment of the invention provides an end-to-end Mongolian-Chinese speech translation method based on comparative learning in order to shorten the distance between Mongolian speech and Chinese text cross-modal data. Firstly, a positive and negative sample pair is constructed based on enhanced data by using a data enhancement method; then, contrast loss is introduced, the similarity between positive sample pairs is maximized, meanwhile, the similarity between negative samples is minimized, and therefore the model is encouraged to learn feature representation with higher discrimination; and finally, fusing the multi-task learning loss and designing the total loss of the model, so that the model can learn more effective speech-text representation mapping, and the accuracy of Mongolian-Chinese speech translation is greatly improved.

The invention discloses a speech translation method and system based on artificial intelligence, and belongs to the technical field of speech data processing. Speech features corresponding to a target speech signal are obtained by extracting data features in the preprocessed target speech signal; recognizing voice features corresponding to the target voice signal by adopting an artificial intelligence model, obtaining first target text information corresponding to the target voice signal, calling third-party text information translation software to translate the first target text information, obtaining second target text information, and sending the second target text information to a server; and finally, broadcasting the second target text information, thereby realizing rapid translation, improving the accuracy of speech translation by means of strong data processing capability of artificial intelligence, reducing the requirement for speech feature extraction, and further improving the effect of speech translation.

The invention provides a recording transmission system applied to an intelligent ring, belongs to the technical field of intelligent rings, and solves the technical problems of single recording function, unstable data transmission, low safety performance and the like of an existing intelligent ring. Comprising a data collection module, a data processing module, a recording content classification module, a self-learning module, a classification storage management module, a remote transmission module, a user interaction module, a cloud server, a power management module, a security privacy module, a remote positioning module, a personalized customization module, an intelligent device control module, an NFC module, a health monitoring module and a static elimination module. An intelligent mobile phone remote control module, a voice translation module and a receiving terminal module. The system has the functions of data collection and processing, content classification, classified learning, classified storage, remote transmission, user interaction, power management, privacy protection, remote positioning, personalized customization, intelligent equipment control, health monitoring, static elimination, intelligent mobile phone remote control and voice translation.

Provided herein is a method for non-audible speech detection and output. The method comprises providing a radio frequency (RF) sensing device configured to be coupled to a head of a user. The method further comprises using the RF sensing device to collect RF signal data associated with movement of one or more speech articulators of the user. The method further comprises outputting or facilitating an output comprising a non-audible speech translation using at least in part processed RF signal data, wherein the non-audible speech of the user comprises continuous speech by the user.

The invention provides a directional pickup method and device, AR glasses, equipment, a medium and a product. The method comprises the following steps: acquiring a plurality of voice signals collected by a plurality of microphones on the AR glasses; performing beam forming according to the plurality of voice signals to obtain voice beams in different directions; target speakers who are speaking currently are recognized according to the voice beams in the different directions, translation results corresponding to the voice beams of the target speakers are output, and the target speakers comprise the user and / or candidate speakers who are dialogue with the user. According to the invention, the pickup accuracy of the target voice can be improved, and the accuracy of voice translation is improved.

The invention belongs to the technical field of mobile terminal communication and language translation, and particularly relates to an intelligent speech translation mobile phone and system capable of realizing multilingual inter-translation, which are characterized in that a two-way speech separation module and unit, a language recognition related module and unit, an end-cloud collaborative translation module and unit and a system are compatible with related module and unit collaboration; two-way voice collection, separation, language recognition and end-cloud collaborative translation are completed in a call scene, the system is compatible with a mainstream mobile operating system and communication application, a resource isolation and process linkage mechanism is adopted to guarantee operation compatibility, multilingual communication obstacles in the call scene are effectively solved, complex environment interference is adapted, the real-time performance and accuracy of translation are considered, and the system is suitable for large-scale popularization and application. And the native function of the terminal does not need to be modified.

Embodiments of the present disclosure relate to a speech translation method, an apparatus, an electronic device, and a medium. The method includes generating a speech representation corresponding to a source-language audio based on the audio. The method also includes obtaining prompt content related to a target language. In addition, the method also includes generating a target-language text corresponding to the audio based on the speech representation and the prompt content.

The invention provides a speech translation method and device, and the method comprises the steps: translating source language speech data based on a speech translation model, and obtaining a target language text; a training target of the speech translation model comprises minimizing a difference between a first target language prediction text generated based on source language sample speech data and a translation label corresponding to the source language sample speech data, and minimizing a difference between speech features of the source language sample speech data and text features of a first source language sample text. And minimizing the difference between a second target language prediction text generated based on the pseudo source language speech features and a translation label corresponding to the second source language sample text. According to the method and the device, the problem of scarcity of annotated voice data is solved by efficiently utilizing relatively rich text data in a low-resource language scene, so that the performance of a voice translation model is improved.

The invention provides a multi-mode joint learning-based multi-task speech recognition and translation system, and the system comprises an audio mode encoder which is used for extracting the feature data of an audio mode through a pre-trained audio pre-training model; the video modal encoder is used for extracting feature data of a video modal by using a pre-trained video pre-training model; the audio-video mode encoder is used for carrying out bimodal feature fusion according to the extracted feature data of the audio mode and the extracted feature data of the video mode and extracting bimodal fusion feature data; the decoder is used for receiving the bimodal fusion feature data and converting the bimodal fusion feature data into a target format; and the first fitting output module is used for simultaneously fitting the speech recognition output and the speech translation output according to the joint loss function minimization. Through the multi-modal learning model, the problem of insufficient performance of a single mode in a complex environment is solved, and the accuracy and robustness in speech recognition and speech translation tasks are remarkably improved.

The invention provides a language recognition method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: carrying out the sliding extraction of a voice segment from a to-be-recognized voice according to a sliding window of a preset size, and detecting a voiceprint turning point from the to-be-recognized voice; under the condition that the voiceprint turning point is detected from the voice segment in the first sliding window, carrying out displacement adjustment on the first sliding window according to the voiceprint turning point so as to enable the voiceprint turning point to be located at the end point position of the first sliding window; and performing language recognition on the shifted voice segment in the first sliding window to obtain a language recognition result. According to the scheme, the voiceprint turning point in the sliding window can be detected, the jumping moment of the speaker in the sliding window can be recognized, the sliding window is moved according to the jumping moment to divide the voice segments, language recognition of the voice segments of multiple languages and a single language in the sliding window can be avoided, and the user experience is improved. Therefore, the accuracy of language recognition of the to-be-recognized speech is higher, and the accuracy of speech translation is further improved.

Provided herein is a method for non-audible speech detection and output. The method comprises providing a radio frequency (RF) sensing device configured to be coupled to a head of a user. The method further comprises using the RF sensing device to collect RF signal data associated with movement of one or more speech articulators of the user. The method further comprises outputting or facilitating an output comprising a non-audible speech translation using at least in part processed RF signal data, wherein the non-audible speech of the user comprises continuous speech by the user.

The invention relates to the technical field of voiceprint recognition, in particular to a two-person dominated multi-round inquiry type voice dialogue intelligent recognition method. The method comprises the following steps: S1, erecting a sound collector to obtain the sound of a speaker, and carrying out sound source orientation discrimination; s2, performing speaker identity initial recognition and gender recognition according to the voiceprint features of the speaker voice, and performing sound source orientation correction and fusion to obtain a final speaker identity recognition result; s3, scene language translation is carried out on the speaking content through a speech translation algorithm in combination with the inquiry scene, and a translation result is output; s4, performing rolling type arrangement and integration on the multi-round dialogue content corresponding to the identity; and S5, generating a template medical record according to the sorted and integrated dialogue content. According to the invention, the inquiry process has the remarkable advantages of high efficiency and intelligence, the working efficiency of doctors can be greatly improved, and the efficiency advantage is particularly obvious under the condition of emergency treatment.

The invention discloses an end-to-end speech translation method and translation system capable of controlling duration, which effectively reduces the translation delay by introducing a speech end-to-end scheme. By constructing a brand new token and introducing a token alignment scheme, the length of a translation result is effectively controlled; by controlling the token duration and adding a length control variable, cross-language timbre and rhythm cloning is guided, so that relatively excellent cross-language timbre and rhythm cloning is achieved.

A system for translating speech from at least two source languages into another target language provides direct speech to target language translation. The target text is converted to speech in the target language through a TTS system. The system simplifies speech recognition and translation process by providing direct translation, includes mechanisms described herein that facilitate mixed language source speech translation, and punctuating output text streams in the target language. It also in some embodiments allows translation of speech into the target language to reflect the voice of the speaker of the source speech based on characteristics of the source language speech and speaker's voice and to produce subtitled data in the target language corresponding to the source speech. The system uses models having been trained using (i) encoder-decoder architectures with attention mechanisms and training data using TTS and (ii) parallel text training data in more than two different languages.

Provided herein is a method for non-audible speech detection and output. The method comprises providing a radio frequency (RF) sensing device configured to be coupled to a head of a user. The method further comprises using the RF sensing device to collect RF signal data associated with movement of one or more speech articulators of the user. The method further comprises outputting or facilitating an output comprising a non-audible speech translation using at least in part processed RF signal data, wherein the non-audible speech of the user comprises continuous speech by the user.