Neural Network Text Classification Using Clickstream Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing technologies face challenges in effectively classifying user-generated questions with unseen text or user behavior, particularly in environments like TurboTax, where data is unlabeled and contains ambiguities, misspellings, and incomplete information, limiting the accuracy of text classification.
Innovation Solution
A deep learning neural network system utilizing character and word embeddings, combined with clickstream data, to classify user input questions. This system employs Bi-LSTM models and self-attention mechanisms to extract features from text and clickstream data, enabling the classification of questions into specific categories even without sufficient training examples, using a 'zero shot learning' approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional natural language processing technologies are used to classify user-generated questions, then the system is simpler to implement, but the accuracy of classifying unseen text or user behavior deteriorates due to unlabeled data, ambiguities, misspellings, and incomplete information
Solution Approach 1:
The patent segments the text classification task into multiple processing stages: embedding generation (character and word levels), feature extraction (using Bi-LSTM networks), attention mechanism application (self-attention and cross-attention), and classification. This segmentation allows each component to specialize in handling specific aspects of the complex classification problem, improving overall accuracy while making the system more manageable.
Solution Approach 2:
The patent implements nested neural network structures where Bi-LSTM layers are nested within attention mechanism layers, which are in turn nested within the classification framework. Character embeddings are nested within word embeddings, and multiple attention layers are nested sequentially. This nesting allows hierarchical processing of information at different levels of abstraction.
2Measurement precision
If deep learning neural network systems with multiple embeddings and attention mechanisms are used, then the accuracy of classifying unseen text improves, but the computational resources and time required increase
Solution Approach 1:
The patent pre-computes and stores character embeddings and word embeddings in lookup tables during an offline training phase. When classifying new text, the system simply retrieves these pre-computed embeddings rather than calculating them from scratch, significantly reducing online processing time while maintaining high classification accuracy.
Solution Approach 2:
The Bi-LSTM networks process sequences of embeddings continuously, maintaining hidden states that capture contextual information across the entire input sequence. The attention mechanisms continuously weigh different parts of the input sequence to identify relevant features, enabling efficient processing without requiring multiple discrete analysis passes.
3Measurement precision
If deep learning neural network systems with multiple embeddings and attention mechanisms are used, then the accuracy of classifying unseen text improves, but the computational resources and complexity increase
Solution Approach 1:
The patent employs universal embedding tables that can represent both character-level and word-level information, and attention mechanisms that can operate on different input types (character embeddings, word embeddings, or their combinations). This multi-functionality reduces the need for separate specialized components for each processing level, simplifying the overall system architecture while maintaining high accuracy.
4Ease of operation
If traditional text classification methods are used on unlabeled user-generated content, then the system is easier to operate, but the ability to handle ambiguities, misspellings, and incomplete information deteriorates
Solution Approach 1:
The patent introduces embedding layers as intermediary components that transform raw text (including misspellings and ambiguous words) into numerical vector representations. These embeddings capture semantic meaning and contextual relationships, serving as a bridge between the raw input and the classification decision. The attention mechanisms then act as intermediaries to selectively focus on relevant parts of the embedded representation, handling ambiguities by weighing different features according to their importance.
Data Source
AI summary
A computer-implemented method is provided to perform text classification with a neural network system. The method includes providing a computing device to receive input datasets including user input question text and feed the datasets to the neural network system. The neural network system includes one or more neural networks configured to extract and concatenate character-based features, word-based features from the question datasets and clickstream embeddings of clickstream data to form a representation vector indicative of the question text and user behavior. A representation vector is fed into fully connected layers of a feed-forward network. The feed-forward network is configured to predict a first class and a second class associated with respective user input questions based on the representation vector.