300 results about "Speech translation" patented technology

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

Automatic video dubbing methods and apparatuses automatically dub a video signal into a different target language than the original while controllably preserving vocal characteristics of original speakers in the corresponding audio, based on separating audio from the video, preserving audio timings, separating voice and non-voice segments from background noise, and using ASR, MT and signal processing techniques to generate properly sized and selectable speech translations into different languages corresponding to speaker voices in the video and emotions, voice and / or textual characteristics.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

Disclosed are apparatuses, systems, and techniques that may use machine learning for reversible translations of speech utterances. The techniques include training and using duplex neural networks (NNs) having a first subnetwork and a second subnetwork that are mirror images of each other. Training data for training the duplex NNs may include a target output that includes a first speech utterance in a first language, a first training input that includes the target output distorted by a noise, and a second training input that includes a second speech utterance in a second language. The duplex NNs may be trained to identify, using the first training input and the second training input, at least one of the target output or the first noise.

The invention belongs to the technical field of speech translation, and particularly relates to a video speech automatic translation method based on a pre-training model, and the method comprises the following steps: 1, preprocessing video and audio data; step 2, voice recognition and language detection; step 3, machine translation and text post-processing; step 4, speech synthesis and audio mixing; step 5, synchronizing video processing and subtitles; step 6, quality control and multi-dimensional evaluation; 7, carrying out model iteration and data closed loop; and step 8, system deployment and engineering implementation. Through deep fusion of efficient transfer learning of the pre-training model and the multi-modal technology, a high-precision, low-cost and easy-to-expand video speech translation solution is constructed, the time and labor cost of globalized content production is greatly reduced, the cross-language communication efficiency is improved, immersive multi-language experience is provided, and the method is suitable for popularization and application. And a data-driven continuous optimization mechanism is established, so that the system performance is improved along with the increase of the use scale.

The invention relates to the technical field of artificial intelligence, and discloses a Bluetooth communication intelligent speech translation method and system based on multi-mode enhancement, and the method comprises the steps: collecting a multi-channel audio signal through a built-in multi-microphone array of a Bluetooth device, carrying out the dynamic direction self-adaptive beam forming of the multi-channel audio signal, and carrying out the self-adaptive beam forming of the multi-channel audio signal; extracting a Mel spectrogram feature of the direction enhancement signal, identifying lip regions of a plurality of candidate speakers in each frame of real-time speaking video captured by a camera, performing time sequence convolution on the lip regions to obtain a lip movement time sequence embedded vector, calculating a correlation score with the Mel spectrogram feature, separating the direction enhancement signal, and obtaining a lip movement time sequence embedded vector; and performing text transcription and conversion on the high-confidence separation voice to obtain a translation language text, and sending the synthesized target translation voice to a preset mobile terminal through the Bluetooth device to obtain a target translation result. According to the method, the real-time performance and accuracy of speech translation are improved in a multi-person scene, far-field speech, noise interference and accent difference.

The invention discloses a cross-modal attention, global memory and dynamic convolution-based speech translation model training method and device, and a speech translation method and device, and relates to the technical field of speech processing and machine translation. A speech translation model is designed to comprise a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model decoder, a global memory network, a dynamic convolution decoder and an output layer. The cross-modal attention adapter projects audio features and performs multi-head cross attention fusion with text embedding; the global memory network updates and enhances historical memory on the basis of a gating mechanism and a Transform Encoder; and the dynamic convolution decoder performs multi-scale convolution extraction on the decoded hidden representation and fuses with the memory, so that the translation quality is improved. According to the method, deep fusion of voice and text, continuous memory with contextual coherence and high-quality translation generation can be realized, the end-to-end voice translation performance is remarkably improved, and the actual requirements of real-time and high-quality end-to-end voice translation in a complex scene are met.

The present application provides a model training method and apparatus, a speech-to-speech translation method and apparatus, and a medium. The method includes: obtaining a speech recognition sample and a real speech-to-speech translation sample; generating a pseudo-labeled speech-to-speech translation sample based on the speech recognition sample; and training a speech-to-speech translation model based on the pseudo-labeled speech-to-speech translation sample and the real speech-to-speech translation sample. Therefore, the model training precision can be improved.

The invention relates to an intelligent real-time language synchronous translation system and a terminal thereof. The system comprises a voice segmentation module used for carrying out semantic analysis on target voice to be translated and segmenting the target voice according to obtained semantic information to obtain voice units; the scene recognition module is used for analyzing the voice unit to obtain scene information; the syntactic rhythm analysis module is used for converting the voice unit into a text, performing syntactic analysis, and analyzing the voice unit in combination with the information and the semantic information to obtain voice rhythm information and statement structure information; the semantic optimization module is used for correcting the semantic information based on the statement structure information, the scene information and the voice rhythm information to obtain optimized semantic information; and the speech translation module is used for translating the optimized semantic information based on the target language to obtain a target language translation. According to the system, through multi-module cooperation, voice semantics can be accurately mastered, different scenes are fitted, and the accuracy, smoothness and practicability of translation are effectively improved.

An automatic online voice translation system and method, comprising a virtual translator, a SIP gateway, and a storage, wherein the SIP gateway receives a call from a caller, when the virtual translator is not activated, the SIP gateway establishes a communication between the caller and the agent by streaming in real-time caller's voice directly to the agent and agent's voice directly to the caller, and when the virtual translator is activated, the SIP gateway creates audio files based on the received voice streams of both the caller and the agent, and sends the created audio files to the virtual translator for translating the caller's voice into a language understood by the agent, before transmitting the voice to the agent. In another embodiment, a contact center platform is used for applying features such as call recording, Interactive Voice Response (IVR) service, and conferencing service to the call.

The embodiment of the invention provides a speech translation method and device, equipment and a storage medium. The method comprises the following steps: in response to a first audio clip in a source language, converting the first audio clip into first audio embedding; in response to determining that the at least one audio clip is obtained before the first audio clip and / or at least one translation clip in a target language corresponding to the at least one audio clip, constructing a first model input based on the first audio feature embedding, at least one audio embedding corresponding to the at least one audio clip, and at least one translation embedding corresponding to the at least one translation clip; and using the trained machine learning model to generate a first translation segment in the target language corresponding to the first audio segment based on the first model input.

The invention discloses a voice translation system based on a Bluetooth headset, which relates to the technical field of voice translation of Bluetooth headsets and comprises a voice acquisition module, a voice speed detection module, a rhythm evaluation module, a voice regulation and control module and a text verification module. The rhythm evaluation module is used for extracting rhythm characteristic information in a user voice signal collected by the Bluetooth earphone in the voice speed fluctuation time period, analyzing the rhythm characteristic information and evaluating the intensity of statement rhythm mutation in user expression in the voice speed fluctuation time period; and the voice regulation and control module is used for selecting and executing a corresponding voice processing strategy according to the evaluation result, and carrying out dynamic regulation and control in the execution process. According to the method, the problem that statement rhythm mutation caused by speech speed fluctuation cannot be dynamically regulated and controlled is solved, intelligent adaptation and whole-process response of a speech processing strategy are realized, structural optimization is performed on a translation result, and translation accuracy and semantic coherence are remarkably improved.

The invention provides a speech translation method, device, equipment and product, and is applied to the technical field of artificial intelligence. The speech translation method comprises the steps that speech to be translated and associated data of the speech to be translated are acquired, the associated data comprise associated images and / or associated videos, the associated images comprise image information of at least part of objects described by the speech to be translated, and the associated videos display lip actions when a speaking object expresses the speech to be translated; and based on the to-be-translated speech and the associated data, translating the to-be-translated speech through the multi-modal speech translation model to obtain a translation result of the to-be-translated speech. Therefore, in combination with the images and / or videos related to the voice, end-to-end voice translation is carried out through the multi-mode voice translation model, and the voice translation quality is effectively improved.

The invention discloses a speech translation method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring an audio clip; performing representation extraction on the audio clip to obtain a plurality of initial high-order speech representations of the audio clip, and performing phoneme classification on each initial high-order speech representation to determine an initial phoneme set; matching phonemes in the initial phoneme set by using each proprietary vocabulary phoneme set in a target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio clip; replacing the initial high-order speech representation of the target proprietary vocabulary with a target high-order speech representation to form a high-order speech representation set of the audio clip; and translating the high-order speech table collection to obtain a target translated text of the audio clip. By means of phoneme-level proprietary vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenes in a speech translation task can be relieved.

Embodiments of the present disclosure relate to a method and apparatus for speech translation, an electronic device, and a medium. The method includes obtaining an audio in a source language, where the audio includes a specific type of information. The method further includes obtaining prompt content related to a target language. In addition, the method further includes generating, based on the audio and the prompt content, a target-language text corresponding to the audio, where the target-language text includes a punctuation mark corresponding to the specific type of the information.

The embodiment of the invention belongs to the technical field of artificial intelligence, and relates to a verbal skill recommendation method applied to seat questioning and answering and related equipment, and the method comprises the steps: receiving customer voice data collected by an audio collection terminal; performing voice translation operation on the client voice data to obtain a client voice text; performing semantic analysis operation on the client voice text to obtain client semantic information; reading a verbal skill database, and obtaining recommended verbal skill content corresponding to the customer semantic information from the verbal skill database; and outputting the recommended verbal skill content. According to the method and the device, the seat can be effectively helped to analyze the potential focus of attention of the customer, the scene that the seat is carried by himself or herself is avoided, the communication becomes more natural, efficient and smooth, the seat is helped to improve the communication skill, and meanwhile, the customer experience can be effectively improved.

The invention provides a voice processing method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a target voice and a translation mode of the target voice; based on a voice encoder module and a translation mode in the voice processing model, encoding the target voice to obtain voice content features and acoustic features of the target voice; translating the target voice based on a large language model, a translation mode and voice content features in the voice processing model to obtain a target translation text of the target voice; and based on a voice decoder module, the translation mode and the acoustic features in the voice processing model, performing voice synthesis on the target translation text to obtain a target translation voice of the target voice. According to the invention, adaptive processing is carried out on voice coding, translation and voice decoding through a voice processing model of an integrated framework integrating voice translation and voice simultaneous transmission in combination with a translation mode, so that the deployment cost is better reduced, and the real-time performance and quality of voice processing are optimized.

The invention provides a remote conference real-time voice recognition optimization method and system based on a visual context, and the method comprises the steps: collecting the visual context information according to a projection screen image; acquiring a voice audio and inputting the voice audio into the voice recognition model to obtain a voice recognition result; and adjusting the voice recognition result by using the context information. On the basis of a traditional remote conference voice real-time recognition function, word and image content information of a current projection screen is analyzed, and the information is fused into a voice recognition model, so that words with low recognition probability or ambiguous words are optimized. According to the method, the error recognition rate can be effectively reduced, the vocabulary recognition problem in the professional field is solved, and the overall performance of the real-time speech translation function is improved.

The invention relates to a speech translation method, device and equipment of an end cloud translation system and a medium, and relates to the technical field of intelligent speech translations, the end cloud translation system comprises an audio receiving end and a cloud, the method is applied to the audio receiving end, and the method comprises the following steps: obtaining a first language audio of the audio receiving end; sending the first language audio to a cloud; and receiving and playing a second language audio, wherein the second language audio is translated by the cloud according to the first language audio and then is sent out. The method can be widely applied to cross-border conference scenes, and the cross-language communication efficiency and experience are improved.

The invention provides a voice, picture and subtitle alignment method and system for real-time voice translation synthesis, and relates to the technical field of voice processing. According to the method, millisecond alignment of translated voice and video is realized through fragment-level parallel processing, compared with a traditional scheme, the delay is greatly reduced, and the problem that translated voice pictures are asynchronous during live broadcast is solved; the method comprises the following steps: automatically generating real-time voice translation according to live broadcast content, and respectively processing to obtain a corresponding srt file and a translated m3u8 fragment; therefore, there is no need to manually translate and generate subtitles in advance, and there is no need to replace the audio of the original video in advance, thereby saving the manpower cost, and providing a more universal live broadcast stream subtitle scheme. Besides, when the languages are switched, the player can automatically select the starting time point, so that the method supports multiple playing modes such as online live broadcast and live broadcast playback.

The invention discloses a multi-modal language conversion method and a related product. The input data is coded through a multi-mode coding unit in the multi-mode decoding unit, vector representation of the input data is obtained, the vector representation is decoded through a plurality of decoding units in the multi-mode decoding unit, and target data obtained after language conversion is conducted on the input data is obtained and output. Wherein the plurality of decoding units are respectively obtained by carrying out association training on different basic decoding units, and each decoding unit is related to a multi-modal cross-language understanding function. By adopting the multi-mode coding unit and the multi-mode decoding unit, the problems of information loss and time delay caused by a chained processing mode in a traditional speech translation system are solved, and the balance between real-time performance and accuracy is improved.

The invention discloses a speech translation method, a related device, equipment and a storage medium, and the speech translation method comprises the steps: obtaining a plurality of initial recognition texts of a to-be-translated speech and a thinking chain used for indicating translation; processing the plurality of initial recognition texts according to the thinking chain by utilizing a large language model to obtain a final translation text of the to-be-translated speech; wherein the thinking chain at least comprises the following steps: carrying out error correction based on a plurality of initial recognition texts to obtain a voice recognition text, translating the voice recognition text into a target language to obtain a first translated text, translating the first translated text into a source language of the voice to be translated to obtain a second translated text, and translating the second translated text into a target language; and correcting the first translated text at least based on the difference between the second translated text and the speech recognition text to obtain a final translated text. According to the scheme, the accuracy of realizing speech translation by the large language model can be improved, especially under the condition of data resource limitation.

To provide a system for smoothly proceeding with a meeting among participants who speak different languages.SOLUTION: A system includes a voice recognition section, a translation section, and a voice synthesis section. The voice recognition section performs voice recognition of utterance of a participant in a meeting. The translation section translates utterance recognized by the voice recognition section. The voice synthesis section outputs contents translated by the translation section as voice.SELECTED DRAWING: Figure 1

A speech translation system may provide real-time or near real-time translation of speech uttered by a person or emitted from a media device. The speech translation system may include a device that may receive audio representing speech in a source language and output audio representing speech in a target language. The speech translation system may translate the speech in portions representing semantically cohesive speech segments such that the target speech reflects the semantic meaning of words, phrases, and / or clauses as used in the context of the source speech. The speech translation system may condense the speech segments prior to or during translation to reduce verbosity. The speech translation system may selectively translate some speakers and not others, and may determine voice characteristics of source speech and apply identifying characteristics to the target speech that allow a user to differentiate respective target speech from different speakers based on the identifying characteristics.

The invention provides an end-to-end Mongolian-Chinese speech translation method and system based on comparative learning, a storage medium and electronic equipment, and relates to the technical field of speech translation. The embodiment of the invention provides an end-to-end Mongolian-Chinese speech translation method based on comparative learning in order to shorten the distance between Mongolian speech and Chinese text cross-modal data. Firstly, a positive and negative sample pair is constructed based on enhanced data by using a data enhancement method; then, contrast loss is introduced, the similarity between positive sample pairs is maximized, meanwhile, the similarity between negative samples is minimized, and therefore the model is encouraged to learn feature representation with higher discrimination; and finally, fusing the multi-task learning loss and designing the total loss of the model, so that the model can learn more effective speech-text representation mapping, and the accuracy of Mongolian-Chinese speech translation is greatly improved.

According to the translation device of the intelligent vehicle-mounted system and the control system of the translation device, the vehicle-mounted external translation device providing a multi-language translation service is constructed and generated through the service data terminal and the microphone, a driver and passengers are translated based on the NLP technology, and then a voice assistant of a vehicle-mounted sound control system is awakened by playing translation voice, so that the translation service of the vehicle-mounted system is improved. And the control instruction is issued to the voice assistant to complete vehicle control in a non-default language, so that the intelligent level of the vehicle is enhanced, a driver and passengers can realize part of functions of the vehicle through voice control even under the condition that the language is impassable, the driving safety is remarkably improved through games, and the driving experience is improved. In addition, the translation device can translate and play response voice of the voice assistant, intelligent interactive conversation between the driver and the vehicle-mounted sound control system is achieved, meanwhile, the translation device supports offline and networking work modes, and the voice translation task can be completed even under the condition that no network exists.

The invention discloses a speech translation method and system based on artificial intelligence, and belongs to the technical field of speech data processing. Speech features corresponding to a target speech signal are obtained by extracting data features in the preprocessed target speech signal; recognizing voice features corresponding to the target voice signal by adopting an artificial intelligence model, obtaining first target text information corresponding to the target voice signal, calling third-party text information translation software to translate the first target text information, obtaining second target text information, and sending the second target text information to a server; and finally, broadcasting the second target text information, thereby realizing rapid translation, improving the accuracy of speech translation by means of strong data processing capability of artificial intelligence, reducing the requirement for speech feature extraction, and further improving the effect of speech translation.

The invention belongs to the technical field of voice processing, and relates to an end-to-end voice translation model-based audio translation method and related equipment, and the method comprises the steps: carrying out the encoding of a to-be-processed audio sent by a user terminal through a voice feature encoder, and extracting important feature data; further encoding through a semantic feature encoder, and converting into semantic feature data; the source language decoder decodes the semantic feature data into a source language text, so that the system can understand and process original voice information; the text sequence decoder decodes the semantic feature data by using the source language text to generate detailed text sequence data; and finally, splitting the text sequence data, and generating and outputting an accurate target translation text. According to the invention, through close cooperation of the encoders and the decoders, high-accuracy translation from audios to texts is effectively realized, and the accuracy of translation contents and proper expression of contexts are ensured; in addition, the system can process multiple languages, and the applicability and flexibility of the system are enhanced.

The invention provides a recording transmission system applied to an intelligent ring, belongs to the technical field of intelligent rings, and solves the technical problems of single recording function, unstable data transmission, low safety performance and the like of an existing intelligent ring. Comprising a data collection module, a data processing module, a recording content classification module, a self-learning module, a classification storage management module, a remote transmission module, a user interaction module, a cloud server, a power management module, a security privacy module, a remote positioning module, a personalized customization module, an intelligent device control module, an NFC module, a health monitoring module and a static elimination module. An intelligent mobile phone remote control module, a voice translation module and a receiving terminal module. The system has the functions of data collection and processing, content classification, classified learning, classified storage, remote transmission, user interaction, power management, privacy protection, remote positioning, personalized customization, intelligent equipment control, health monitoring, static elimination, intelligent mobile phone remote control and voice translation.

Provided herein is a method for non-audible speech detection and output. The method comprises providing a radio frequency (RF) sensing device configured to be coupled to a head of a user. The method further comprises using the RF sensing device to collect RF signal data associated with movement of one or more speech articulators of the user. The method further comprises outputting or facilitating an output comprising a non-audible speech translation using at least in part processed RF signal data, wherein the non-audible speech of the user comprises continuous speech by the user.