Paraphrase Generation Using Word and Character Level Attention Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural machine translation systems are limited in predicting target words given unknown source words not included in their training datasets, particularly in bilingual translation and paraphrase generation tasks.
Innovation Solution
A paraphrase generation system utilizing a combination of word-level and character-level attention-based models, including bidirectional LSTM networks with residual connections, and reinforcement learning to generate accurate paraphrases by leveraging embeddings and semantic relationships from knowledge bases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If word-level attention-based models are used for paraphrase generation, then translation speed is improved, but accuracy deteriorates when encountering unknown source words
Solution Approach 1:
The patent segments the word-level model into character-level units, enabling the system to process and predict unknown words by analyzing their character compositions. This segmentation allows the model to maintain speed while improving accuracy for out-of-vocabulary words through character-level attention mechanisms.
Solution Approach 2:
The patent implements a nested architecture where character-level models are integrated within the word-level model framework. The character-level attention-based model operates inside the word-level model, allowing both levels to contribute to the final prediction, thereby resolving the contradiction between speed and accuracy for unknown words.
2Measurement precision
If character-level models are added to word-level models, then prediction accuracy for unknown words is improved, but device complexity increases
Solution Approach 1:
The patent merges the character-level and word-level models into a unified architecture where both levels share computational resources and processing frameworks. By combining these models rather than running them separately, the system achieves improved accuracy for unknown words while minimizing the increase in overall device complexity.
Data Source
AI summary
The present disclosure pertains to a paraphrase generation system. The system comprises one or more hardware processors and/or other components. The system is configured to obtain a training corpus. The training corpus comprises language and known paraphrases of the language. The system is configured to generate, based on the training corpus, a word-level attention-based model and a character-level attention-based model. The system is configured to provide one or more candidate paraphrases of a natural language input based on both the word-level and character-level attention-based models. The word-level attention-based model is a word-level bidirectional long short term memory (LSTM) network and the character-level attention-based model is a character-level bidirectional LSTM network. The word-level and character level LSTM networks are generated based on words and characters in the training corpus. In some embodiments, the LSTM networks are stacked residual LSTM networks comprising residual connections between stacked layers of a given LSTM network.