Small Language Model Two-Phase Training Data Curation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high cost and energy consumption of training, deploying, and maintaining large language models (LLMs) pose significant challenges, along with concerns about the controllability and governability of these models.
Innovation Solution
A small language model (SLM) is generated through a two-phase training process, using a filtered general dataset and a synthetic dataset to achieve capabilities similar to LLMs with significantly fewer parameters and tokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large language models are trained with trillions of parameters and tokens to achieve high capability, then model performance is improved, but training cost and energy consumption increase substantially
Solution Approach 1:
The patent creates synthetic training data that copies the essential knowledge patterns and linguistic structures from large language models, enabling small language models to learn from compressed representations rather than raw corpora. This copying approach transfers critical capabilities while avoiding the energy-intensive processing of massive datasets.
Solution Approach 2:
The patent fundamentally changes the parameter scale from trillions to billions or millions, while compensating through targeted data curation and synthetic dataset generation. This parameter reduction is achieved by transforming the training approach from scale-dependent to quality-dependent, using carefully constructed datasets that maximize learning efficiency.
2Reliability
If large language models are trained with massive scale parameters, then model capability is improved, but deployment and maintenance cost increase
Solution Approach 1:
The patent employs smaller, less expensive model architectures that can be deployed on commodity hardware rather than requiring specialized infrastructure. These smaller models are more accessible and can be deployed across distributed systems, reducing centralized maintenance burdens and enabling broader adoption.
3Reliability
If large language models are used to achieve high capability, then performance on reasoning tasks is improved, but controllability and governability decrease
Solution Approach 1:
The patent segments the training process into distinct phases: pre-training on filtered general data for foundational knowledge, then fine-tuning on synthetic data for specific reasoning capabilities. This segmentation allows independent optimization and control of different skill sets, making the model more manageable and controllable.
Solution Approach 2:
The patent implements feedback loops during synthetic data generation and model evaluation, where model performance on reasoning tasks informs subsequent data generation and training adjustments. This feedback mechanism enables continuous optimization and better alignment with desired capabilities while maintaining controllability.
Data Source
AI summary
Systems and methods for generating a small language model are provided. In particular, a computing device may obtain a general dataset including a plurality of general data, annotate a subset of the general dataset based on one or more classifier metrics indicative of a quality of the general dataset, train a classifier based on the annotated subset of the general dataset and the one or more classifier metrics, analyze each general data of the general dataset to determine a score for each of the one or more classifier metrics associated with the respective general data using the trained classifier, generate a filtered general dataset by filtering the general dataset based on one or more filters, train the small language model with the filtered general dataset, generate a synthetic dataset for refining the small language model, and train the small language model with the synthetic dataset.