LLM Training Data Filtration Using Tokenization Quality Ratios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training data filtration for Large Language Models (LLMs) are inadequate as they often fail to effectively filter out low-quality documents, leading to suboptimal model performance and resource inefficiencies.
Innovation Solution
A method involving tokenization and statistical distribution analysis, using tokens-per-byte and tokens-per-character ratios to identify and remove low-quality documents, ensuring the filtered data maintains the desired distribution and quality for LLM training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional AI techniques use large data sets to train LLMs, then the model can learn patterns and draw conclusions, but low-quality documents in the data set lead to suboptimal model performance
Solution Approach 1:
The patent applies preliminary action by performing data filtration and quality assessment before the actual model training process. The system computes tokens-per-byte and tokens-per-character ratios, positions documents in statistical distributions, and removes low-quality documents beforehand, ensuring that only high-quality data enters the training phase, thus improving model performance without requiring excessive data volume
Solution Approach 2:
The patent utilizes parameter changes by transforming raw training data into a quality-assessed dataset through computational metrics. By calculating tokens-per-byte and tokens-per-character ratios and using statistical distribution positioning, the system changes the parameter quality of training data, converting unfiltered large datasets into refined datasets with optimized characteristics for effective model training
2Reliability
If existing filtration methods are used, then some data processing is performed, but they fail to effectively filter out low-quality documents
Solution Approach 1:
The patent applies parameter changes by introducing specific computational metrics (tokens-per-byte ratio, tokens-per-character ratio) and statistical distribution positioning to assess and filter data quality. These parameter transformations enable effective identification of low-quality documents through quantitative analysis rather than simple heuristic filtering
Solution Approach 2:
The patent replaces mechanical or simple rule-based filtration systems with a computational analysis system. Instead of using basic keyword filtering or manual quality assessment, the system employs automated computation of statistical ratios and distribution positioning to objectively evaluate and filter documents, achieving higher reliability through algorithmic substitution
3Productivity
If low-quality documents are not filtered out, then all available data is used for training, but computational resources are wasted and retraining is needed
Solution Approach 1:
The patent applies the taking out principle by extracting and removing low-quality documents from the training dataset through systematic filtration. By computing quality metrics and identifying documents that fall below statistical thresholds, the system extracts harmful elements (low-quality data) from the whole, preventing waste of computational resources during training and eliminating the need for costly retraining
Solution Approach 2:
The patent performs preliminary filtration action before model training to prevent computational resource waste. By assessing data quality in advance using tokens-per-byte and tokens-per-character ratios, the system prepares a refined dataset that maximizes training efficiency from the outset, avoiding the need for retraining due to poor data quality
Data Source
AI summary
An embodiment samples raw data pertaining to a domain to form sample domain data, which includes a set of documents. A first pre-tokenization attribute of a first document in the set is computed. The first document is tokenized to generate a first set of tokens corresponding to the first document. A first post-tokenization attribute of the first document is computed. The first document is positioned in a statistical distribution of the set of documents according to a ratio of a pre-tokenization attribute and a post-tokenization attribute of each document in the set. a subset of documents including the first document is selected, from the statistical distribution such that each member of the subset has a value of the corresponding ratio below a first threshold configured for the distribution. The subset is filtered out from the sample domain data to form filtered data. A model is trained using filtered data.