Long-Term Memory Network for Multi-Word Text Comprehension

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional question answering approaches fail to fully comprehend the meaning of text information and are limited in generating multi-word answers, requiring large amounts of data and high supervision.

Innovation Solution

A Long-Term Memory Network (LTMN) architecture that combines an End-to-End Memory Network with a Long Short-Term Memory (LSTM) network, enabling minimal supervision and generating multi-word answers by encoding input sentences and questions into vector representations and using a soft attention mechanism to match relevant factoids.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional question answering approaches (IR-based or knowledge-based) are used, then the system can process queries efficiently, but the system cannot fully comprehend the meaning of text information

Engineering Contradiction:

Improvetext comprehension capabilityVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent combines end-to-end memory networks with LSTM networks to create a hybrid architecture that merges the strengths of both approaches. The memory network component handles information retrieval and matching, while the LSTM component provides sequential processing and contextual understanding, together achieving full text comprehension without requiring separate traditional IR or knowledge-based modules

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The proposed LTMN architecture serves multiple functions within a single unified system: it performs question encoding, memory storage, attention-based retrieval, and answer generation. This multi-functional design eliminates the need for separate specialized systems for different question answering tasks, achieving comprehensive text understanding through a versatile neural network framework

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If deep neural network architectures like end-to-end memory networks are used, then the system can understand and memorize text information, but the system requires a large amount of data and high supervision

Engineering Contradiction:

Improvetext understanding capabilityVSAvoidtraining data requirement

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent implements preliminary action by pre-training the model on available data to learn question encoding and memory formation patterns before actual question answering. The external memory is pre-populated with encoded sentences, and the attention mechanisms are pre-configured to recognize relevant information patterns, reducing the need for extensive supervised training data during deployment

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses copying by storing encoded representations of sentences in external memory rather than requiring the model to re-process original text during inference. The memory network copies relevant information from the input text into structured memory slots, allowing the system to work with compressed representations that require less training data while maintaining understanding capabilities

Inventive Principle:

Principle #26Copying

3Ease of operation

If end-to-end memory networks are used, then the system can be trained with minimal supervision, but the system is limited to single-word answers

Engineering Contradiction:

Improvetraining supervision requirementVSAvoidanswer generation capability

Core Design Contradiction:

Ease of operationVSAdaptability or versatility

Solution Approach 1:

The patent transitions from single-word answer generation to multi-word answer generation by adding a temporal dimension through LSTM networks. The LSTM processes sequences of words over time steps, allowing the system to generate answers with multiple words while maintaining the end-to-end training framework. This dimensional extension transforms the output space from discrete single tokens to sequential multi-token responses

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

4Adaptability or versatility

If the system uses external memory and LSTM modules, then the system can generate multi-word answers, but the system complexity increases

Engineering Contradiction:

Improvemulti-word answer generationVSAvoidnetwork architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the complex LTMN architecture into distinct functional modules: question encoding module, memory storage module, attention mechanism module, and LSTM answer generation module. Each module has a specific responsibility and can be independently optimized or replaced. This segmentation manages complexity by creating manageable components while maintaining their integrated functionality for multi-word answer generation

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10963789B2Long-term memory networks for knowledge extraction from text and publications

Publication Date: 2021.03.30 CONDUENT BUSINESS SERVICES LLC

AI summary

A long-term memory network method and system for text comprehension. A recurrent neural network can be provided, which includes an external memory module and a long-short term memory unit, wherein said recurrent neural network encodes raw text information into vector representations, forms memories, finds relevant sentences to answer questions, and generates multi-word answers to said questions utilizing the long short term memory unit.