Semi-supervised Deep Learning for Customer Support Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing preprocessing techniques for textual data in customer-support systems are inefficient due to the need for manual specification of regex rules and the difficulty in obtaining labeled training data, leading to inadequate handling of noisy data and limited scalability across languages and sources.
Innovation Solution
A semi-supervised deep-learning approach that uses a combination of recurrent neural networks, feed-forward networks, and Sparsemax to encode and translate customer request sentences into importance probabilities, extracting relevant sentences and forming embeddings to select agent responses, while utilizing unlabeled training data and contrastive loss for model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If model-based machine-learning techniques are used to preprocess textual data, then system performance is improved, but labeled training data is difficult and time-consuming to obtain
Solution Approach 1:
The patent performs preliminary action by using unlabeled data to pretrain the language model before fine-tuning with labeled data. This preliminary pretraining step allows the model to learn general language patterns and representations from abundant unlabeled text, reducing the amount of time and resources needed to obtain and process labeled training data while still achieving high system performance.
Solution Approach 2:
The patent uses unsupervised fine-tuning that copies and adapts the pretrained model to the specific task domain without requiring extensive labeled data. The unsupervised fine-tuning process creates task-specific representations by leveraging the structure and patterns in the data itself, effectively copying the general language understanding to the specific customer support domain without manual labeling.
2Reliability
If supervised models are used to achieve reasonable performance, then a substantial amount of training data is required, but obtaining and processing such data increases complexity and time
Solution Approach 1:
The patent performs preliminary action by using unlabeled data to pretrain the language model before fine-tuning with labeled data. This preliminary pretraining step allows the model to learn general language patterns and representations from abundant unlabeled text, reducing the amount of time and resources needed to obtain and process labeled training data while still achieving high system performance.
Solution Approach 2:
The patent uses unsupervised fine-tuning that copies and adapts the pretrained model to the specific task domain without requiring extensive labeled data. The unsupervised fine-tuning process creates task-specific representations by leveraging the structure and patterns in the data itself, effectively copying the general language understanding to the specific customer support domain without manual labeling.
3Ease of manufacture
If regex rules are used for preprocessing, then the process is deterministic and easy to implement, but the complexity grows quickly and rules are limited by what can be explicitly defined
Solution Approach 1:
The patent replaces the mechanical system of manual regex rule creation with an intelligent system based on pretrained language models. Instead of manually defining and maintaining complex regex patterns, the system uses neural networks that have been pretrained on large corpora to automatically understand and process textual data, replacing the manual, rule-based approach with an automated, learning-based approach that scales better.
Solution Approach 2:
The patent changes the fundamental parameter of how preprocessing is performed - from static, manually-defined regex rules to dynamic, learned representations from pretrained models. The model's parameters are adjusted during unsupervised fine-tuning to capture task-specific patterns, allowing the system to adapt to different data sources and languages without manually creating new rules for each case.
Data Source
AI summary
During operation, the system receives a customer request. Next, the system segments the customer request into customer request sentences. The system then encodes each sentence from the customer request with information sequentially collected from the previously observed sentences. Next, the system translates the encodings to sparse probabilities that measure the importance of sentences from the customer request. The system then extracts relevant sentences from the customer request based on the importance. Next, the system forms an extracted-sentence customer request embedding from embeddings for the extracted relevant customer request sentences. The system then uses the extracted-sentence customer request embedding to select an agent response from a set of possible agent responses based on comparisons between the extracted-sentence customer request embedding and embeddings for the set of possible agent responses. Finally, the system presents the selected agent response to the customer to facilitate resolving the customer request.