Neural Network Embedding for Noncontiguous Word Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to effectively obtain embedding vectors for noncontiguous word sequences, such as idioms, as they do not distinguish between varying lengths and treat overlapping sequences independently, leading to high similarity between sequences with different meanings.
Innovation Solution
A learning device that constructs a neural network with multiple hidden layers defined for various arrangement patterns, including noncontiguous word patterns, to learn and output embedding vectors for sets of words, allowing for the representation of noncontiguous words and distinguishing between sequences of different lengths and meanings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a neural network is trained to predict surrounding words for a given word or word sequence, then embedding vectors can be obtained for contiguous words, but the system cannot effectively represent noncontiguous word sequences such as idioms
Solution Approach 1:
The patent segments the input word sequence into multiple possible arrangements by introducing arrangement patterns. Each pattern defines a specific structure (e.g., contiguous words, noncontiguous words with gaps). The neural network is trained separately for each arrangement pattern, allowing it to specialize in capturing the relationships characteristic of each pattern type. This segmentation enables the system to handle both contiguous and noncontiguous word sequences effectively.
Solution Approach 2:
The patent adds a new dimension to the embedding space by incorporating arrangement pattern information. Instead of a single embedding vector for each word sequence, the system generates multiple embedding vectors corresponding to different arrangement patterns. This dimensional expansion allows the system to distinguish between sequences that differ in their arrangement structure, thereby improving the precision of semantic representation.
2Productivity
If overlapping word sequences are treated independently, then the neural network can process each sequence separately, but sequences with different lengths and meanings become indistinguishable
Solution Approach 1:
The patent segments the processing task by creating separate neural network models or processing branches for each arrangement pattern. Each segment is specialized for a specific pattern type (e.g., bigrams, trigrams, noncontiguous patterns). This segmentation allows independent processing of each pattern type while maintaining the ability to distinguish between them, as each segment preserves the structural characteristics of its designated pattern.
Solution Approach 2:
The patent applies local quality by assigning different processing characteristics to different arrangement patterns. Each arrangement pattern receives tailored processing parameters, attention mechanisms, or network architectures that are optimized for its specific structure. For example, contiguous patterns may use standard convolutional operations, while noncontiguous patterns may use attention mechanisms that account for gaps. This localized optimization ensures that each pattern type is processed with appropriate precision.
3Ease of manufacture
If traditional embedding methods are used for word sequences, then processing is simple and fast, but idiomatic expressions and noncontiguous word patterns cannot be properly captured
Solution Approach 1:
The patent segments the complex task of representing all word sequences into multiple simpler subtasks, each handled by a specialized neural network component trained on a specific arrangement pattern. This segmentation makes the overall system more manageable and interpretable, as each component has a clear, focused function. The modular structure also allows for easier implementation and training compared to a single monolithic model attempting to handle all patterns simultaneously.
Data Source
AI summary
A learning device includes one or more processors. The processors input, to an input layer of a neural network including hidden layers defined for respective first arrangement patterns indicating arrangement of one or more words, and output layers connected with some of hidden layers, one or more first morphemes conforming to any of first arrangement patterns, among morphemes included in a document, and learn the neural network to minimize a difference between one or more second morphemes conforming to any of second arrangement patterns indicating arrangement of one or more words, among morphemes included in the document, and output morphemes from the neural network for the input first morphemes. The processors output an embedding vector of the first morphemes that is obtained based on a weight of the learned neural network.