Optimized BERT Intent Recognition with Linear Non-Linear Feature Extraction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional intent recognition methods, such as those using LSTM and TextCNN, struggle with incorporating pre-training knowledge and have poor knowledge reasoning capabilities, while BERT models require extensive processing time and fail to extract sufficient knowledge information, especially with a large number of labels, leading to inefficiencies and reduced accuracy.

Innovation Solution

The proposed solution optimizes and trains a BERT model to enhance intent recognition by preprocessing data to extract both linear and non-linear features, using an optimized BERT unit with reduced encoder layers and additional normalization and transformation layers to improve feature extraction and recognition accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a conventional BERT model is used for intent recognition, then knowledge reasoning capability is improved, but processing time increases significantly

Engineering Contradiction:

Improveknowledge reasoning capabilityVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the feature extraction process into linear feature extraction and non-linear feature extraction pathways. The linear pathway uses optimized BERT with reduced encoder layers for fast processing, while the non-linear pathway processes only when linear features are insufficient, thereby reducing overall processing time while maintaining knowledge reasoning capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by using only the necessary portion of the BERT model (optimized with reduced encoder layers) for linear feature extraction, and selectively applying non-linear feature extraction only when needed based on confidence thresholds, avoiding full model processing for all inputs.

Inventive Principle:

Principle #16Partial or excessive action

2Productivity

If conventional intent recognition models (LSTM, TextCNN) are used, then processing speed is maintained, but knowledge reasoning capability and recognition accuracy deteriorate

Engineering Contradiction:

Improveprocessing speedVSAvoidrecognition accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent merges multiple feature extraction approaches (optimized BERT for linear features, non-linear transformation layers for non-linear features) into a unified intent recognition model, combining the speed advantages of conventional models with the knowledge reasoning capabilities of BERT to achieve both high processing speed and high recognition accuracy.

Inventive Principle:

Principle #5Merging (Combining)

3Reliability

If a BERT model with many parameters is used, then knowledge representation capability is improved, but model complexity and training time increase

Engineering Contradiction:

Improveknowledge representation capabilityVSAvoidmodel complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts and utilizes only the essential knowledge representation capabilities of BERT through optimized encoder layers, removing redundant complexity while retaining the core knowledge reasoning functionality needed for intent recognition.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent changes key parameters of the BERT model, specifically reducing the number of encoder layers while adjusting other parameters to maintain knowledge representation capability, thereby reducing model complexity and training time while preserving essential functionality.

Inventive Principle:

Principle #35Parameter changes

4Adaptability or versatility

If BERT model is used with large quantity of labels, then coverage of intent categories is improved, but recognition accuracy deteriorates due to insufficient knowledge extraction

Engineering Contradiction:

Improveintent category coverageVSAvoidrecognition accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent adds a non-linear feature extraction dimension to complement the linear feature extraction from optimized BERT. This additional dimension enables the model to extract sufficient knowledge information for multi-label scenarios, improving recognition accuracy across a large quantity of intent categories.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20230343327A1Intent recognition methods, apparatuses, and devices

Publication Date: 2023.10.26 ALIPAY (HANGZHOU) INFORMATION TECH CO LTD

AI summary

Intent recognition is described. Obtained data to be recognized is preprocessed to obtain target data. Feature extraction processing is performed on the target data based on an intent recognition model to obtain a linear feature and a non-linear feature of the target data, where the intent recognition model is obtained by optimizing and training a bidirectional encoder representations from transformers (BERT) model. Intent recognition processing is performed based on the linear feature, the non-linear feature, and the intent recognition model to obtain an intent recognition result of the obtained data to be recognized.