Knowledge Graph Linearization for Flexible Language Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pipeline-based language model training suffers from error propagation, intermodular dependencies, lack of flexibility, and resource-intensive complexity, limiting its ability to handle complex tasks and adapt to new data types.
Innovation Solution
A method and system for transforming and structuring a knowledge graph using multi-hop linearization and masked language modeling to train a language model, enabling it to handle multimodal data and complex tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If pipeline-based language model training is used, then sequential processing steps can be defined for NLP tasks, but error propagation occurs and system reliability deteriorates
Solution Approach 1:
The patent merges multiple pipeline processing steps into a unified transformer model architecture. Instead of separate sequential modules (tokenization, parsing, named entity recognition), all functions are integrated into a single model that processes input through multiple attention mechanisms in parallel, eliminating error propagation between stages while maintaining comprehensive NLP capabilities
Solution Approach 2:
The transformer model serves multiple NLP functions simultaneously through its universal architecture. The same model performs tokenization, parsing, named entity recognition, and other language processing tasks by configuring different attention mechanisms and training objectives, rather than requiring separate specialized modules for each function
2Ease of manufacture
If pipeline-based processing is used, then each step can be optimized independently, but intermodular dependencies increase and device complexity worsens
Solution Approach 1:
Multiple independently optimizable modules are merged into a single transformer architecture where each function (tokenization, parsing, entity recognition) is implemented through different attention mechanisms within the same model. This maintains the ability to optimize each function while eliminating the complexity of coordinating multiple independent modules and their interdependencies
Solution Approach 2:
The transformer model segments processing into parallel attention mechanisms that operate simultaneously rather than sequentially. Different attention heads handle different linguistic functions in parallel, achieving modular optimization benefits while maintaining unified system architecture that reduces overall complexity
3Ease of manufacture
If pipeline-based approaches are used, then fixed processing routines can be established, but adaptability to new tasks decreases and ease of operation worsens
Solution Approach 1:
The transformer model transitions from fixed pipeline routines to a dynamic architecture where attention mechanisms can be configured and trained for different tasks. The model adapts to new NLP tasks through configurable attention patterns and training data, maintaining stability through its unified architecture while enabling versatility across diverse language processing applications
Solution Approach 2:
The universal transformer architecture handles multiple NLP tasks through the same underlying model by configuring different attention mechanisms and training objectives. This single model replaces multiple fixed pipelines, providing both stability through unified processing and versatility through task-specific configuration capabilities
4Reliability
If separate models are developed for each pipeline step, then each step can be specialized, but resource consumption increases and productivity decreases
Solution Approach 1:
Multiple specialized models for different NLP steps are merged into a single transformer model that performs all functions simultaneously. This integration maintains the specialized capabilities of individual models while achieving superior training efficiency through unified architecture, reducing redundant computations and resource consumption
Solution Approach 2:
The transformer model serves as a universal system that performs tokenization, parsing, named entity recognition, and other NLP tasks that previously required separate specialized models. This multi-functionality maintains the reliability benefits of specialization while dramatically improving productivity through efficient resource utilization and reduced training overhead
Data Source
AI summary
A method of enhancing language model performance through structured knowledge injection performed by a computing system including a memory and a processor including obtaining knowledge base data including a predetermined knowledge graph, generating linearly structured data by structuring the obtained knowledge base data into a text format, training a first language model based on the generated linearly structured data, and providing a predetermined application service based on the trained first language model. The generating linearly structured data includes generating the first linearly structured data by structuring the knowledge graph in the text format based on multi-hop linearization.