ML-NLP Classifier Multi-Step Training for Domain Content

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Natural language processing (NLP) models struggle to effectively classify domain-specific content due to differences in language patterns and terminologies compared to general content, leading to suboptimal performance in modeling language patterns and identifying specific topics.

Innovation Solution

A multi-step training approach for machine-learning NLP classifiers, starting with a general corpus, followed by domain-specific corpus training, and fine-tuning with a topic-specific corpus annotated with labels, to generate a domain-specific language model that can accurately classify content into specific topics.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a language model is trained using unsupervised machine-learning from general content, then the model can perform various natural language processing tasks, but the model cannot be optimized for modeling natural language in domain-specific content

Engineering Contradiction:

Improvegeneral NLP task capabilityVSAvoiddomain-specific language modeling accuracy

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The training process is segmented into three distinct stages: general corpus training for foundational NLP capabilities, domain-specific corpus training for domain language patterns, and topic-specific corpus fine-tuning for precise topic classification. This segmentation allows each stage to optimize for its specific objectives without compromising the others.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The general corpus training serves as a preliminary action that establishes the foundation for NLP tasks before specialized domain-specific training occurs. This preliminary general training ensures the model has broad capabilities before narrowing focus to specific domains and topics.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If a language model is trained on domain-specific content, then the model can model domain-specific language patterns, but the model cannot effectively classify content into specific topics of interest

Engineering Contradiction:

Improvedomain-specific language modeling accuracyVSAvoidtopic classification precision

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The training is segmented into domain-specific language modeling stage and topic-specific classification stage. The domain-specific corpus training teaches the model domain language patterns, while the subsequent topic-specific corpus fine-tuning with labeled data enables precise topic classification, with each stage building upon the previous one.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The topic-specific corpus includes labeled data that provides feedback signals during fine-tuning. This labeled information enables the model to learn accurate topic classifications by comparing predictions against ground truth labels, continuously improving classification precision through the feedback loop.

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If a language model is trained on general content, then the model has broad language understanding, but the model performs suboptimally on domain-specific topic classification

Engineering Contradiction:

Improvegeneral language understandingVSAvoiddomain-specific topic classification precision

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The training process is divided into general language understanding stage using general corpus, domain-specific adaptation stage using domain-specific corpus, and topic-specific fine-tuning stage using topic-specific corpus. Each stage builds upon the previous one, progressively improving precision for specific domains and topics while maintaining general understanding capabilities.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

General corpus training serves as the preliminary action that establishes broad language understanding foundations. This preliminary training ensures the model has versatile NLP capabilities before subsequent specialized training narrows the focus to improve domain-specific and topic-specific classification precision.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11907672B2Machine-learning natural language processing classifier for content classification

Publication Date: 2024.02.20 REFINITIV US ORGANIZATION LLC

AI summary

Computer-readable media, systems and methods may improve classification of content based on a machine-learning natural language processing (ML-NLP) classifier. The system may train a general language model based on a general corpus, further train the general language model based on a domain-specific corpus to generate a domain-specific language model, and conduct supervised machine-learning based on the domain-specific language using topic-specific corpus labeled as relating to topics of interest to generate the ML-NLP classifier. Accordingly, the ML-NLP classifier may be trained on a general corpus, further trained on a domain-specific corpus, and fine-tuned on a topic-specific corpus. In this manner, domain-specific content may be classified into topics of interest. The ML-NLP classifier may classify content into the topics of interest.