Transformer Classifier Joint Intent Entity Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine learning systems for generating responses to user input, such as dialogue, often produce limited and irrelevant outputs due to exposure bias and lack of diversity, which are not adequately addressed by existing techniques like scheduled sampling and maximum mutual information criteria.
Innovation Solution
The use of transformer-based machine classifiers that can jointly model multiple tasks without requiring joint labeling, employing loss masking and attention mechanisms to improve processing efficiency and output diversity, allowing for the training of a single machine classifier to perform various natural language understanding tasks like sentence classification and question answering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional neural network architectures are used for multi-task learning, then joint labeling is required to model multiple tasks, but this increases data annotation complexity and processing time
Solution Approach 1:
The patent segments the multi-task learning process into independent task-specific output sequences, where each task is modeled separately through its own output sequence. This allows the system to handle multiple tasks without requiring joint labeling, as each task can be trained independently while sharing the same encoder representation.
Solution Approach 2:
The patent implements a universal encoder that serves multiple tasks simultaneously. The single encoder processes input sequences and generates representations that can be used for various downstream tasks (classification, sequence labeling, etc.) without requiring separate encoders or joint labeling schemes, thereby achieving multi-functionality with a unified architecture.
2Measurement precision
If multiple separate machine classifiers are trained for different tasks, then task-specific performance is optimized, but processing time and computational resources increase
Solution Approach 1:
The patent merges multiple task-specific classifiers into a single unified machine classifier architecture. The shared encoder processes all input sequences, and task-specific outputs are generated through separate output layers for each task. This consolidation reduces processing time and computational resources compared to training and running multiple separate classifiers, while maintaining task-specific performance through dedicated output sequences.
3Reliability
If existing techniques like scheduled sampling and maximum mutual information criteria are applied, then exposure bias is partially addressed, but output diversity and relevance remain limited
Solution Approach 1:
The patent employs loss masking as a feedback mechanism during training, where the loss function selectively masks certain positions in the output sequence based on task-specific requirements. This provides targeted feedback to the model, improving both training stability and output diversity by guiding the model to focus on relevant positions for each task while maintaining overall training reliability.
Data Source
AI summary
Systems described herein may use transformer-based machine classifiers to perform a variety of natural language understanding tasks including, but not limited to sentence classification, named entity recognition, sentence similarity, and question answering. The exceptional performance of transformer-based language models is due to their ability to capture long-term temporal dependencies in input sequences. Machine classifiers may be trained using training data sets for multiple tasks, such as but not limited to sentence classification tasks and sequence labeling tasks. Loss masking may be employed in the machine classifier to jointly train the machine classifier on multiple tasks simultaneously. The user of transformer encoders in the machine classifiers, which treat each output sequence independently of other output sequences, in accordance with aspects of the invention do not require joint labeling to model tasks.