Enterprise Language Model Training With Balanced Data Blending
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in effectively processing multi-modal inputs that combine natural language and structured data due to the complexity of interpreting natural language, leading to increased resource utilization and minimal predictive performance improvements.
Innovation Solution
A domain-specific language model is trained using a balanced training dataset that combines enterprise and domain-specific text partitions, generating semantically dense embeddings to complement structured inputs, thereby improving predictive performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If natural language inputs are used in machine learning models, then the model can process unstructured text data, but the complexity of interpreting natural language increases resource utilization and limits effectiveness
Solution Approach 1:
The patent introduces an enterprise-specific language model as an intermediary component between raw natural language text and the predictive machine learning model. This language model processes and transforms unstructured text into structured representations (embeddings, extracted entities, relationships) that are easier for the predictive model to interpret, thereby reducing the complexity burden while maintaining natural language processing capability
Solution Approach 2:
The system segments the natural language processing task into distinct components: (1) enterprise-specific language model training phase, (2) text feature extraction phase, and (3) predictive modeling phase. This segmentation allows each component to be optimized independently, reducing overall system complexity while maintaining versatility
2Reliability
If multi-modal inputs combining natural language with structured data are used, then predictive performance may improve, but processing and memory resources increase significantly
Solution Approach 1:
The patent extracts only the most relevant and non-redundant features from natural language text using the enterprise-specific language model. Instead of feeding entire text documents or all possible text features to the predictive model, the system extracts key entities, relationships, and semantic representations, thereby reducing resource consumption while maintaining predictive performance
Solution Approach 2:
The system transforms natural language text from its original high-dimensional, unstructured form into compressed semantic representations (embeddings) with optimized dimensionality. This parameter transformation reduces the computational burden on the predictive model while preserving the essential information needed for accurate predictions
3Loss of information
If traditional language modeling is used to extract text sequences, then text features can be obtained, but enterprise-level language models are lacking and much information is redundant with structured data
Solution Approach 1:
The patent implements a preliminary action by training an enterprise-specific language model before the predictive modeling phase. This pre-trained language model captures domain-specific terminology, relationships, and patterns that are unique to the enterprise context, ensuring that subsequent text feature extraction is both informative and non-redundant with respect to structured data
Solution Approach 2:
The system incorporates feedback mechanisms where the enterprise-specific language model is trained using labeled data that reflects the actual relationships between text features and target predictions. This feedback loop ensures that the language model learns to extract features that are genuinely useful for prediction while avoiding redundancy with structured data inputs
Data Source
AI summary
Various embodiments of the present disclosure provide a language model training technique. The language model training technique may include a data blending preprocessing step to improve the performance of the language model at an enterprise level. The data blending technique includes receiving an enterprise data partition from a plurality of enterprise data partitions associated with an enterprise data source, receiving a domain-specific data partition from a plurality of domain-specific data partitions associated with one or more domain data sources that are different than the enterprise data source, storing the enterprise data partition as an initial training partition of a plurality of balanced training partitions within a balanced training dataset, and generating a balanced training partition by appending a portion of the domain-specific data partition to the initial training partition. A domain-specific language model may then be trained based on the balanced training dataset.