Class-Aware Word Splitter Training for Versatile NLP Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing (NLP) systems face challenges in optimizing tokenization patterns to enhance downstream model performance without limiting the versatility of the tokenizer, especially when the downstream model's parameters and architecture are not assured.
Innovation Solution
A method for training a word splitter that adjusts tokenization based on biasedness of occurrence frequency in multiple classes, optimizing the tokenizer to enhance performance in downstream models by using a weighted score that considers class information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If tokenization is optimized for downstream model performance, then inference quality improves, but tokenizer versatility deteriorates
Solution Approach 1:
The patent implements dynamic tokenization by training the word splitter to adaptively adjust splitting patterns based on input characteristics and class information. The system dynamically selects optimal tokenization strategies rather than using fixed patterns, allowing the tokenizer to maintain versatility across different tasks while optimizing for specific downstream model performance when needed.
Solution Approach 2:
The patent changes the parameters of the word splitter by training it with class information and occurrence frequency data. This modifies the tokenization behavior to be more informed and adaptive, improving inference quality without completely sacrificing versatility. The parameter adjustments allow the system to balance between optimized performance and general applicability.
2Ease of manufacture
If downstream model parameters are fixed, then training simplicity improves, but performance optimization deteriorates
Solution Approach 1:
The patent performs preliminary action by training the word splitter with class information and occurrence frequency data before downstream model training. This pre-training step incorporates domain-specific knowledge into the tokenization process, improving performance optimization without complicating the overall training pipeline. The word splitter is prepared in advance to handle class-specific tokenization needs.
Solution Approach 2:
The patent introduces class information and occurrence frequency data as intermediaries between the raw input and the tokenization process. These intermediaries guide the word splitter to produce more informative tokens that improve downstream model performance. The intermediaries act as a bridge that enhances performance without requiring changes to the downstream model architecture or parameters.
Data Source
AI summary
A method including: in training a word splitter with a training data set including data pieces each associating a letter string data piece and a class data piece representing one of classes that the letter string data piece pertains with each other, the word splitter outputting split letter string data including letter strings obtained by splitting an inputted letter string data piece, the split letter string data serving as input data to be inputted into a machine-learning model that performs an inference process, training the word splitter based on biasedness of occurrence frequency in the classes for each of a first letter strings included in pieces of the split letter string data being obtained by inputting the letter string data piece into the word splitter and by splitting the letter string data piece in respective different splitting patterns corresponding to the letter string data piece.