Pre-training LSTM Layers for Document Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training document classification neural networks are inefficient, requiring extensive computational resources and labeled data, while struggling with stability and generalization.
Innovation Solution
The approach involves pre-training LSTM layers on unlabeled data through sequence autoencoding or language modeling tasks, followed by fine-tuning on labeled data to improve stability and generalization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional training methods are used for document classification neural networks, then the network can be trained on labeled data, but the training requires extensive computational resources and time
Solution Approach 1:
The patent applies preliminary action by pre-training the LSTM layers on unlabeled data before the main supervised training phase. This pre-training initializes the network weights with useful representations from large unlabeled corpora, so that subsequent fine-tuning on labeled data requires less time and computational resources while achieving better classification accuracy.
2Reliability
If traditional training methods are used for document classification neural networks, then the network can be trained on labeled data, but extensive computational resources are required
Solution Approach 1:
The patent applies preliminary action by pre-training the LSTM layers on unlabeled data before the main supervised training phase. This pre-training initializes the network weights with useful representations from large unlabeled corpora, so that subsequent fine-tuning on labeled data requires less time and computational resources while achieving better classification accuracy.
3Reliability
If traditional training methods are used, then the network trains on labeled data directly, but the network struggles with stability and generalization
Solution Approach 1:
The patent applies preliminary action by pre-training the LSTM layers on unlabeled data before the main supervised training phase. This pre-training initializes the network weights with useful representations from large unlabeled corpora, so that subsequent fine-tuning on labeled data requires less time and computational resources while achieving better classification accuracy.
Solution Approach 2:
The patent applies self-service by having the network learn useful representations from unlabeled data through self-supervised pre-training tasks. The network serves itself by extracting meaningful features from the structure and patterns in unlabeled text, reducing its dependence on large amounts of manually labeled training data.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes training a language model neural network, wherein the language model neural network comprises the one or more neural network layers and an output layer, and wherein training the language model neural network comprises determining pre-trained values of the parameters of the one or more neural network layers from initial values of the parameters of the one or more neural network layers; and training the neural network on a plurality of training documents to determine trained values of the parameters of the one or more neural network layers from the pre-trained values of the parameters of the one or more neural network layers.