Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

210 results about "Machine translation" patented technology

Machine translation, sometimes referred to by the abbreviation MT (not to be confused with computer-aided translation, machine-aided human translation (MAHT) or interactive translation) is a sub-field of computational linguistics that investigates the use of software to translate text or speech from one language to another.

Artificial intelligence speech recognition system

The invention discloses an artificial intelligence speech recognition system, and the system comprises a multi-modal feature extraction module which employs an improved Conformer architecture to synchronously extract the time-frequency features and text embedding vectors of speech signals; the joint training module is used for performing joint optimization on ASR and NMT loss functions through an adversarial training strategy, learning voice recognition and machine translation tasks at the same time through joint training, and completing direct mapping from voice features to a target language; the context perception translation engine is used for integrating an attention mechanism of a pre-training language model, carrying out deep coding on the extracted speech features and generating cross-language semantic representation; the self-adaptive post-processing module is used for dynamically optimizing an output result by adopting a reinforcement learning framework, dynamically adjusting the output result according to a reward function, and optimizing translation quality and a speech synthesis effect; the dynamic language recognition module is a real-time language classifier based on a Wave2Vec 2.0 framework and is used for recognizing the language of the input voice in real time; and the incremental field adaptation module is used for quickly updating a field term library by using a LoRA fine tuning technology.
Owner:ANKANG UNIV

Data augmentation using machine translation capabilities of language models

Disclosed are embodiments for improving training data for machine learning (ML) models. In an embodiment, a method is disclosed where an augmentation engine receives a seed example, the seed example stored in a seed training data set; generates an encoded seed example of the seed example using an encoder; inputs the encoded seed example into a machine learning model and receives a candidate example generated by the machine learning model; determines that the candidate example is similar to the encoded seed example; and augments the seed training data set with the candidate example.
Owner:VERIZON PATENT & LICENSING INC

Machine translation method and apparatus, device and storage medium

A machine translation method includes: obtaining first target language text by performing first translation on source language text using an initial NMT model; identifying an untranslated part in the source language text based on the source language text and the first target language text; obtaining an adjusted NMT model by increasing an attention weight corresponding to the untranslated part in the initial NMT mode; and obtaining second target language text by performing second translation on the source language text using the adjusted NMT model.
Owner:BEIJING BAIDU NETCOM SCI & TECH CO LTD

Video machine translation method and device fusing multi-modal fine-grained information

The invention provides a video machine translation method and device fusing multi-modal fine-grained information. The multi-modal fine-grained information fused video machine translation method comprises the following steps: performing information extraction on a picture in a video to obtain fine-grained visual information in the picture, and performing information extraction on an audio in the video to obtain fine-grained audio information in the audio; fusing the fine-grained visual information and the fine-grained audio information into a source text to obtain a fused text; the source text is subtitles to be translated in the video; and inputting the fused text into a machine translation model to obtain a target translation text. According to the video machine translation method and device fusing the multi-modal fine-grained information, the fine-grained visual information and the fine-grained audio information of the video are fused into the source text, machine translation is performed based on the fused text, and the translation accuracy is improved.
Owner:INST OF AUTOMATION CHINESE ACAD OF SCI

PDF optimization translation method and system based on intelligent text detection

The invention provides a PDF (Portable Document Format) optimized translation method based on intelligent text detection, which comprises the following steps of: 1, analyzing a PDF document, respectively identifying a text region and an image region in the PDF document, and extracting a native text block of the text region in the PDF document and structured position information of the native text block; the recognition result and the native text block are combined into a unified structured data set; 3, performing machine translation by adopting a window translation strategy in combination with the context, and extracting a target translation; adjacent text blocks are dynamically associated during translation, and a target text range is defined through boundary markers; and 4, dynamically optimizing the text layout according to the target translation length, and backfilling a translation result to an original position. According to the method, the translated text can be accurately and seamlessly backfilled to the corresponding position of the original image, meanwhile, the content of the original image is ensured not to be shielded, and visual disorder or information loss caused by text coverage is avoided.
Owner:WUHAN UNIV

OWS earphone simultaneous interpretation method based on ASR algorithm

The OWS earphone simultaneous interpretation method based on the ASR algorithm comprises the following five steps: S1, generating a voice feature signal after noise reduction through a distributed multi-microphone array; s2, inputting the voice characteristic signal into a dynamic path planning module based on a knowledge graph to generate a multilingual optimization signal; s3, using the optimized signal to control the lightweight hybrid ASR model, and generating a text sequence signal with a path mark through end-cloud cooperative processing; s4, converting the text sequence signal into a target language text through a neural machine translation module; and S5, receiving a user feedback signal through a multi-modal interaction interface, and updating the path weight and the semantic association parameter updating signal in the knowledge graph by using a reinforcement learning mechanism. According to the OWS earphone simultaneous interpretation method based on the ASR algorithm, the problems of low speech recognition precision and high multilingual simultaneous interpretation delay of an open earphone in a complex sound field environment can be solved.
Owner:SHENZHEN HUAJUE COMM CO LTD

Multi-language corpus automatic construction and translation optimization system

The invention relates to a natural language processing and multi-language data processing technology, and discloses a multi-language corpus automatic construction and translation optimization system which comprises a corpus collection module, a language recognition and grouping module, a semantic alignment module, a translation optimization module and a corpus quality evaluation and screening module. According to the system, multi-language text data can be automatically collected from the Internet, language recognition and structured storage are carried out, semantic vector coding and alignment matching are carried out on sentences of different languages through a cross-language pre-training model, and high-quality parallel corpus generation is achieved. Meanwhile, the translation model is subjected to incremental training and language proportion regulation and control by utilizing the aligned corpora, so that the translation performance is improved; corpus quality is automatically evaluated and screened through a scoring mechanism, and data reliability is guaranteed. The system can be widely applied to the fields of multi-language machine translation, cross-language information retrieval, intelligent corpus construction and the like.
Owner:EAST CHINA JIAOTONG UNIVERSITY

Real-time glasses system and device based on neural network machine translation

The invention relates to the field of translation glasses, in particular to a real-time glasses system and device based on neural network machine translation. The technical problem to be solved by the invention is to provide accurate voice acquisition and visual text positioning to ensure translation coherence in a complex scene. The neural network machine translation-based real-time glasses system and device can realize real-time response, multi-language mixed processing and privacy security. The invention discloses a real-time glasses system and device based on neural network machine translation. The real-time glasses system comprises a beam forming microphone array, a micro transparent display screen, a bone conduction loudspeaker, a front camera, an AI processor, an end-to-end neural network processing assembly line, a dynamic context caching module, a user sight tracking module, a privacy protection module and a user interaction module. According to the invention, accurate voice acquisition and visual text positioning are achieved, and translation coherence in a complex scene is ensured; and the effects of real-time response, multi-language mixed processing and privacy security are realized.
Owner:CHANGCHUN INST OF ELECTRONIC TECH

Simultaneous interpretation device, simultaneous interpretation system, simultaneous interpretation processing method, and non-transitory computer readable storage medium

Provided is a simultaneous interpretation system that performs machine translation processing and speaker identification processing in real time. In the simultaneous interpretation system, the segment processing unit of the simultaneous interpretation device performs high-speed and highly accurate segment processing to obtain sentence data and also obtains a time range in which a word sequence included in the sentence data was uttered, thus, allowing for performing machine translation processing and speaker identification processing in real time. In other words, in the simultaneous interpretation system, the machine translation processing unit performs machine translation processing on sentence data obtained through high-speed and highly accurate segment processing, and performs, in parallel, processing for predicting a speaker who spoke during the period specified by the time range data based on the inputted video stream and the time range data, thus allowing for performing machine translation processing and speaker identification processing in real time.
Owner:NAT INST OF INFORMATION & COMM TECH

Embedded translate, summarize, and auto read

A method of facilitating consumption of online content includes receiving source text for a source article to be translated, the source text being in a source language. The source language for the source text and the target language to which the source text is to be translated are each identified. The source text, the source language, and the target language are each provided to a machine translation model which automatically generates translated text in the target language from the source text. The translated text is provided as input to a generative language model which generates summary text in the target language from the translated text. The summary text is provided to a text-to-speech model which generates summary audio from the summary text. The summary text and summary audio are then sent to a user interface via which the summary text is displayed, and playback of the summary audio is enabled.
Owner:MICROSOFT TECHNOLOGY LICENSING LLC

Method for training machine translation model for generating pseudo parallel translation data, method for obtaining pseudo parallel translation data, and method for training machine translation model

Provided is a pseudo parallel translation data generation apparatus for generating pseudo parallel translation data for accurately performing machine translation in an adaptation target domain even when there exists no parallel translation data for the adaptation target domain. Using other-domains parallel translation data D0(L1-L2), other-domains first language data D0(L1), other-domains second language D0(L2), adaptation target domain first language data D0(R1), and adaptation target domain second language data D0(R2), the pseudo parallel translation data generation apparatus 100 performs optimization processing for a cross-lingual language model including an input data embedding unit 2 and an XLM processing unit 3, and performs parameter optimization processing for a pseudo parallel translation data generation NMT model including the input data embedding unit after the optimization processing and a machine translation processing unit 5. Performing processing using the pseudo parallel translation data generation machine translation model obtained by the parameter optimization processing allows for obtaining pseudo parallel translation data for the adaptation target domain for which no parallel translation data sets exist.
Owner:NAT INST OF INFORMATION & COMM TECH

Cross-border logistics single-multi-language machine translation method based on natural language processing

The invention discloses a cross-border logistics document multi-language machine translation method based on natural language processing, and relates to the technical field of machine translation, and the method comprises the steps: collecting original data of a cross-border logistics document, and carrying out text extraction and preprocessing through an OCR technology; performing language recognition and format analysis on the preprocessed document text to determine a source language and a target language; using a pre-trained cross-language alignment translation model to translate the document text into a target language text; the translation result is input into a compliance auditing module, and automatic compliance auditing is conducted through a knowledge graph and a double-check algorithm; and generating an output containing a translation result and a compliance audit report, and supporting manual review and revision. According to the method, the translation accuracy of the documents can be effectively improved, the manual auditing burden is reduced, and the globalization requirements of cross-border e-commerce and logistics enterprises are met.
Owner:QINGDAO UNIV OF TECH +1

Translation quality assurance based on large language models

A method, computer program product, and computer system are provided for assuring quality of machine translations based on large language models. Data corresponding to an input in a first language is received for machine translation. The received data is translated from the first language to a second language through a translation engine. A confidence value associated with the translated data is determined. The translation is revised with executable source code based on the confidence value being greater than a threshold value. Otherwise, the translation is revised based on sending a prompt to a large language model based on the confidence value being less than the threshold value.
Owner:INTERNATIONAL BUSINESS MACHINE CORPORATION

Machine translation method and system based on large model fusion task disassembly and cognitive planning

The invention discloses a machine translation method and system based on large model fusion task disassembly and cognitive planning. The method comprises the following steps: 1) decomposing a translation task into a plurality of stages, including syntactic analysis, semantic alignment, term precise mapping, context coherence optimization and culture adaptation; a cue word used in the semantic alignment stage prompts a model to accurately recognize semantic connotation of a source language sentence, and finds a most matched semantic expression in a target language; and 2) in each stage, guiding the large model to execute a corresponding translation task by using a specific cue word so as to ensure the translation quality. According to the method, through a multi-stage cognitive translation instruction framework, a context-aware multi-round dialogue engine and an interpretable translation decision tracing mechanism, the quality and efficiency of machine translation are remarkably improved, good adaptive capacity and interpretability are achieved, and the method is suitable for translation requirements of various complex texts.
Owner:GUANGZHOU BINGO SOFTWARE

Intelligent glasses independent communication method and system based on RedCap and eSIM, and medium

The invention provides an intelligent glasses independent communication method and system based on RedCap and eSIM and a medium, and the method comprises the steps: after intelligent glasses are started, initiating a registration request to a 5G core network based on an embedded eSIM chip, and after the core network completes identity authentication and authentication, distributing an independent IP address for the intelligent glasses; collecting interaction data, and encrypting the interaction data; performing data processing on the interaction data based on a cloud AI platform to generate result data; analyzing the result data based on voice recognition and a machine translation model to obtain AR content or an output text or a voice result; the signal intensity of the 5G core network is monitored in real time, the signal intensity is compared with a set intensity threshold value, and a communication mode is dynamically switched based on a comparison result; the intelligent glasses are completely independently networked, free movement and use of the intelligent glasses are achieved, real-time translation, AR navigation, remote cooperation and cloud cooperation application are supported, and user experience and scene adaptability are improved.
Owner:SHENZHEN ORANGE ELECTRONICS CO LTD

A machine translation method applied to a stable deep machine translation model

The present invention discloses a machine translation method applied to a stable deep machine translation model. The steps are as follows: during the training process of the deep model, when performing each backpropagation calculation according to the input source language sentence X, two matrices μ and σ are set for the gradient matrix G of the parameters in the encoder part of the deep model; the matrices μ and σ are used to represent the mean and standard deviation of the gradient matrix G respectively, and the values of the matrices μ and σ are calculated based on the gradient matrix G; during each backpropagation process during model training, the gradient matrix G is normalized using the matrices μ and σ; the usage stage and position of this normalization are controlled through a preset strategy and prior knowledge to perform fast and accurate training of the machine translation model, and finally improve the translation quality of the output translation during decoding. The present invention can make the training process more stable, is not limited to specific models and tasks, has good generality, and makes the effects of the model easier to reproduce on different devices.
Owner:XIAONIU FANYI

A low-power edge simultaneous interpretation system based on audio-text synchronization and visual feature fusion

This invention discloses a low-power edge-side simultaneous interpretation system and method based on audio-text synchronization and visual feature fusion, relating to the fields of multimodal human-computer interaction and machine translation technology. The system achieves high-precision audio-text timing alignment through an audio-text synchronization matching module, generates a lightweight visual feature stream using a visual feature processing module, and performs spatiotemporal fusion by a multimodal fusion inference module. Combined with edge-side heterogeneous computing power scheduling and dynamic power consumption control, it significantly reduces the power consumption of edge devices while ensuring low-latency translation of ≤20ms per frame. This invention fills the technological gap in edge-side low-power multimodal simultaneous interpretation and can be widely applied in edge scenarios such as mobile office, international communication, and smart wearables.
Owner:宋伟光

Translation method and system based on semantic energy conservation network, medium, equipment and program product

The invention provides a translation method and system based on a semantic energy conservation network, a medium, equipment and a program product, and belongs to the technical field of machine translation. In the technical scheme, vocabularies of input source language sentences are expressed in the form of energy, from the perspective of energy conservation, a Hamiltonian deep neural network HDNN and a VI Verlet integrator are used as an energy evolution network for processing evolution of vocabulary position and active quantity information, the relative position and the active quantity after evolution are input into a Transform-based translation network, and a translation result is obtained. The total Hamiltonian energy of the sentences is recalculated through the energy evaluation network, and then the energy difference is calculated to evaluate the translation loss. According to the method, the idea of Hamiltonian dynamics and the advantages of deep learning are combined, evolution is carried out through semantic energy conservation, vocabulary information evolution and feature extraction are carried out through HDNN, the stability and consistency of information are guaranteed, the model can better understand the context relation of an input text, and translation accuracy can be improved.
Owner:INNER MONGOLIA NORMAL UNIVERSITY

Continuous casting machine translation control system and continuous casting machine

The present invention relates to the field of billet continuous casting production equipment, and provides a continuous casting machine translation control system and a continuous casting machine. The continuous casting machine translation control system includes a first hydraulic cylinder, a second hydraulic cylinder, a first hydraulic lock, a second hydraulic lock, a first rodless cavity oil supply pipeline, a second rodless cavity oil supply pipeline, a first rod cavity oil supply pipeline, a second rod cavity oil supply pipeline, a control system, a main control mechanism, and an auxiliary control mechanism. The control system is configured to control the main control mechanism to move the first hydraulic cylinder and the second hydraulic cylinder, and is configured to receive signals from a first position sensor and a second position sensor. When the signals indicate that the moving speeds of the two hydraulic cylinders are inconsistent, an instruction is sent to the auxiliary control mechanism to make the moving speeds of the two hydraulic cylinders consistent. The continuous casting machine includes the above-mentioned translation control system. The continuous casting machine translation control system provided by the present invention can improve the problem of the asynchronous moving speeds of the two hydraulic cylinders, and the continuous casting machine provided with the control system has good stability.
Owner:SGIS SONGSHAN CO LTD

Translation hardware acceleration platform and method based on SGRU

The invention discloses a translation hardware acceleration platform and method based on an SGRU, and relates to the technical field of machine translation, and the platform comprises an accelerator which comprises a control module, an input and output module, an Embedding layer, a distributed pulse cache module, a model parameter cache module, a calculation module and a gating calculation core; the input and output module is respectively connected with the control module, the Embedding layer, the model parameter cache module and the PC terminal; the calculation module is respectively connected with the control module, the Embedding layer, the distributed pulse cache module and the model parameter cache module; the gating calculation core is connected with the control module and the distributed pulse cache module. A calculation heterogeneous design of a calculation module and a gating calculation core is adopted, data conflicts are avoided through distributed caching, dynamic loading and updating are carried out based on a model parameter caching module, overall control is carried out according to an SGRU-based Seq2Seq model, and high-performance and low-power-consumption translation can be realized.
Owner:GUANGDONG UNIV OF TECH

Knowledge learning method and device of a machine translation model

The application provides a knowledge learning method and device of a machine translation model, comprising: constructing a migration dataset; respectively using a machine translation teacher model and a machine translation student model to translate the original text of each sample of the migration dataset to obtain the first candidate word probability distribution and the second candidate word probability distribution of each sample of the migration dataset; determining the migration loss of each sample of the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample of the migration dataset; and realizing the knowledge learning of the machine translation student model to the machine translation teacher model based on the first candidate word probability distribution of each sample of the migration dataset and the migration loss. The application enables the machine translation student model to learn complementary knowledge from the machine translation teacher model through the migration loss, realizes the knowledge accumulation of the machine translation student model, and further improves the translation performance of the machine translation student model.
Owner:TSINGHUA UNIVERSITY

Neural Network Machine Translation Method, Model and Model Formation Method

The present invention relates to a neural network machine translation method, a model, and a model formation method. The method for forming a neural network machine translation model includes: forming an encoder, which includes a first multi-head attention model; forming a decoder, which includes a second multi-head attention model and a future information model, and the future information model represents the fusion of the first attention hidden layer representation of the current predicted word and the already generated words and the second attention hidden layer representation of the current predicted word and the future possible words; forming a first machine translation model through the encoder and the decoder; and performing decoding training on the first machine translation model from left to right and from right to left for the source language sequence to form a neural network machine translation model, wherein the first multi-head attention model and the future information model provide inputs for the second multi-head attention model. The present invention solves the problem that future information cannot be fully utilized when predicting the current word during the process of machine translation.
Owner:THE BOEING CO +1

A biological data processing method and device, computer equipment and storage medium

The application discloses a biological data processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring atomic attribute information of a first atom contained in a first molecular substance and acquiring atomic attribute information of a second atom contained in a second molecular substance; inputting the atomic attribute information of the first atom and the atomic attribute information of the second atom into a machine translation model; in the machine translation model, generating a first reaction intensity value between the first atom and each atom in an atomic set and a second reaction intensity value between the second atom and each atom; the atomic set comprises the first atom and the second atom; and predicting a substance affinity between the first molecular substance and the second molecular substance according to the first reaction intensity value between the first atom and each atom and the second reaction intensity value between the second atom and each atom. The application can improve the accuracy of the obtained substance affinity.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Method for knowledge distillation, apparatus, electronic device, medium and computer program product

The present disclosure provides a method for knowledge distillation, an apparatus, an electronic device, a medium and a computer program product. The method includes: acquiring a training source text and a standard translation text corresponding to the training source text; inputting the training source text into a teacher translation model and a student translation model separately, to obtain a teacher distribution output by the teacher translation model and a student distribution output by the student translation model; obtaining a standard translation distribution according to the standard translation text and the training source text; and performing iterative training on the student translation model according to the teacher distribution, the student distribution, and the standard translation distribution, to obtain a target machine translation model.
Owner:BEIJING YOUZHUJU NETWORK TECH CO LTD +1

A machine translation software defect detection method based on combined semantics

The application discloses a machine translation software defect detection method based on combined semantics, comprising the following steps: S1, obtaining different Chinese translation sentences of the same English source sentence from different translation software; S2, using a word alignment model to correspond English words in the source sentence with segmented Chinese words in the Chinese translation sentences; S3, using a sentence compression model and a syntactic structure analysis method to obtain a main part and each additional part of the English source sentence respectively and form a sub-sentence set; S4, aligning each part of the English source sentence obtained in the step S3 with the corresponding translation part to obtain aligned translation of the main part and the additional part of the source sentence; and S5, detecting errors, including semantic similarity calculation and synonym checking.
Owner:TIANJIN UNIV

Training method of language model for multilingual translation and translation method

The disclosure provides a language model training method and a translation method for multilingual translation, and relates to the technical field of artificial intelligence, in particular to the fields of natural language processing, deep learning, machine translation and the like. The language model training method for multilingual translation comprises: obtaining a sample text pair, the sample text pair comprising a first text in a first language and a second text in a second language, the semantics of the first text being the same as the semantics of the second text; inputting the first text and the second language into a language model to obtain a first translated text in the second language output by the language model; inputting the second text and the second language into the language model to obtain a second translated text in the second language output by the language model; determining a loss value of the language model based on a first difference between the first translated text and the second text and a second difference between the first translated text and the second translated text; and adjusting parameters of the language model based on the loss value.
Owner:BEIJING BAIDU NETCOM SCI & TECH CO LTD

A game-based Mongolian neural machine translation method

A game-based Mongolian neural machine translation method converts problematic factors that affect translation decoding in Mongolian neural machine translation into interference noise in the form of pseudo data. A game model based on a generative adversarial network is constructed based on a noise interference strategy. The model consists of a generator G, an anti-interference device AJ, and a discriminator D. The model generalizes to the interference noise in the three-module game, improving the translation performance of the game model against this noise. During game training, the REINFORCE strategy is used to optimize the parameters of G to obtain training rewards for original data and noise data. The original corpus and pseudo corpus containing interference noise are trained using an adversarial training mechanism. AJ is used to determine the training priority of the noise data based on the judgment of the training data, so that G and D have sufficient decoding capabilities for the added interference noise. Finally, the trained G is used as the translation model for Mongolian translation. This invention can improve the accuracy and efficiency of Mongolian translation.
Owner:INNER MONGOLIA UNIV OF TECH

System and method for active learning based multilingual semantic parser

Described is a system and method for training a multilingual semantic parser. A method includes receiving, by a multilingual semantic parser, a multilingual training dataset, wherein the multilingual training dataset includes pairs of utterances and meaning representations from at least one high-resource language and at least one low-resource language and wherein the multilingual training dataset is initially a machine-translated dataset, training, the multilingual semantic parser, by translating the utterances in the multilingual training dataset to a target language; and iteratively performing selecting, by an acquisition functions estimator, a subset of the multilingual training dataset for human translation, updating the multilingual training dataset with the human-translated subset of the multilingual training dataset with, and retraining, the multilingual semantic parser, with the updated multilingual training dataset.
Owner:OPENSTREAM INC

Deep learning-based ancient modern text machine translation method

The invention discloses an ancient modern text machine translation method based on deep learning, and belongs to the technical field of natural language processing. The invention provides a multi-task collaborative optimization framework aiming at the problems of domain knowledge deficiency, frequent word activation, corpus scarcity and the like of an existing neural network model in ancient language translation. Precise domain knowledge injection is realized by constructing a closed ancient language parallel corpus retrieval library and combining historical term alignment and logic chain reasoning; a probability weighted part-of-speech embedding mechanism is adopted, and grammar feature distribution of virtual word ambiguity and word type activity is dynamically fused, so that the grammar analysis robustness is improved; and semantic generation, retrieval alignment and part-of-speech constraint tasks are jointly optimized through a fixed weight strategy. According to the method, the semantic accuracy and the logic continuity of ancient language translation are improved, and extended application of complex ancient language styles is supported. Experimental results verify that the performance of the method is improved in terms of BLEU and ROUGE-L indexes, and efficient technical support is provided for ancient book digitization and cross-time explanation.
Owner:ZHONGBEI UNIV

A two-stage visual fusion multi-modal translation method and system

The application provides a kind of two-stage visual fusion multimodal translation method and system, to solve the problem of insufficient utilization of visual information in existing multimodal translation, single fusion mechanism, model prone to ignore visual information.The method comprises: obtaining source language text and corresponding image, extracting fine-grained regional visual features and coarse-grained global visual features;In the first stage, the fine-grained regional features are spliced with the text representation and encoded through guided self-attention to realize cross-modal alignment;In the second stage, the coarse-grained global features are projected and broadcasted through gated residual modulation to inject scene-level prior information;The multimodal machine translation target and the visual conditional mask language modeling target are jointly optimized for training.The application effectively improves the translation quality through two-stage multi-granularity fusion, and the BLEU and METEOR indicators on Multi30K and other data sets are significantly better than existing methods, which can be applied to multimedia translation, cross-language retrieval and other fields.
Owner:HENAN NORMAL UNIV