Sentence Embedding via Hybrid Interaction Representation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current sentence embedding models suffer from one-way semantic representation, leading to partial semantic loss due to limited interaction between words, which affects the accuracy and efficiency of text classification tasks.

Innovation Solution

A two-level interaction representation method is introduced, comprising a local interaction representation (LIR) and a global interaction representation (GIR), which are combined to generate a hybrid interaction representation (HIR) to improve sentence embeddings, utilizing LSTM cells and attention mechanisms to capture both local and global interactions between words.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If a one-way action model is used for sentence embedding, then the model structure is simple, but semantic information is lost

Engineering Contradiction:

Improvemodel structureVSAvoidsemantic information

Core Design Contradiction:

Device complexityVSLoss of information

Solution Approach 1:

The patent applies bidirectional encoding by introducing two separate encoders: a forward encoder that processes text from left to right, and a backward encoder that processes text from right to left. This inversion of the traditional unidirectional approach allows the model to capture contextual information from both directions, preventing semantic information loss while maintaining a manageable structure through parallel processing.

Inventive Principle:

Principle #13The other way round (Inversion)

Solution Approach 2:

The patent introduces an intermediary mechanism in the form of bidirectional hidden state interactions. The forward and backward encoders exchange contextual information through shared weight matrices and combined hidden states, acting as intermediaries that integrate semantic information from both directions without requiring a completely complex architectural overhaul.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of information

If a bidirectional interaction model is used for sentence embedding, then semantic information is preserved, but the model complexity increases

Engineering Contradiction:

Improvesemantic informationVSAvoidmodel structure

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent segments the encoding process into distinct forward and backward components, each with its own hidden state sequence and weight matrices. This segmentation allows the complex bidirectional processing to be broken down into manageable, independent modules that can be trained and processed separately, reducing overall model complexity while preserving semantic information.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges the forward and backward encoder outputs at the hidden state level, combining contextual information from both directions into a unified representation. This merging strategy allows the model to capture comprehensive semantic information while avoiding the need for separate processing paths throughout the entire architecture, thus controlling complexity.

Inventive Principle:

Principle #5Merging (Combining)

3Productivity

If traditional embedding models are used, then computational efficiency is maintained, but text representation accuracy is limited

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidtext representation accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent performs preliminary bidirectional encoding to generate comprehensive contextual representations before final sentence embedding is computed. By pre-processing text through both forward and backward passes and storing intermediate hidden states, the model prepares rich semantic information in advance, improving representation accuracy without significantly impacting final computational efficiency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes key parameters including the introduction of bidirectional weight matrices, modified hidden state dimensions, and adjusted learning rates for the dual-encoder architecture. These parameter changes enable the model to achieve higher representation accuracy while maintaining computational feasibility through optimized training configurations and efficient memory utilization.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11436414B2Device and text representation method applied to sentence embedding

Publication Date: 2022.09.06 NAT UNIV OF DEFENSE TECH

AI summary

The invention discloses a device and text representation method applied to sentence embedding, which has determining a parent word and a child words set corresponding to the parent word, obtaining an hidden interaction state of parent word based on the hidden interaction state of all child words; obtaining parent word sequence corresponding to the parent word and obtaining hidden state sequence corresponding to the hidden state of parent word; obtaining the interaction representation sequence of each parent word and other parent word based on the hidden state sequence, generating sentence embeddings. The invention proposes to realize sentence embeddings through a two-level interaction representation. The two-level interaction representation is local interaction representation and a global interaction representation respectively, and combines the two-level representation to generate a hybrid interaction representation, which can improve the accuracy and efficiency of sentence embeddings and be significantly better than the Tree-LSTM model in terms of accuracy.