Span-Level Attention LLM for Multi-Section Knowledge Graph Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional large language models (LLMs) struggle to interpret deep semantic relationships in complex multi-section natural language documents, limiting their effectiveness in converting unstructured text into structured knowledge graphs.
Innovation Solution
A modified LLM architecture with a semantic chunking model is employed to identify data entity tokens, generate text span embeddings, and apply span attention vectors to create structured knowledge graphs, enabling accurate detection and interpretation of semantic relationships across document sections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional LLMs are used to process complex multi-section documents, then they can extract data from unstructured text, but they fail to interpret deep semantic relationships and convert information accurately into structured representations
Solution Approach 1:
The document is divided into multiple text spans, and each span is processed independently through embedding and attention mechanisms. This segmentation allows the model to focus on local semantic relationships within each span while maintaining the ability to connect spans through attention vectors, thereby improving semantic interpretation accuracy without overwhelming the model with entire document complexity
Solution Approach 2:
Attention vectors are introduced as intermediary elements between text spans and the final structured representation. These attention vectors capture the semantic relationships between spans and serve as mediators that bridge the gap between raw text and structured knowledge graph, enabling accurate conversion without requiring the model to process the entire complex document structure at once
2Reliability
If traditional LLMs process multi-section documents, then they can handle unstructured text, but they cannot accurately detect semantic relationships between segments distributed across multiple sections
Solution Approach 1:
The model transitions from processing text in a single dimension to processing it across multiple dimensions: text spans are embedded into a vector space, attention vectors are computed across spans, and these attention vectors are then used to generate structured representations. This multi-dimensional approach enables the model to detect semantic relationships between distributed segments that would be impossible in traditional single-pass processing
Solution Approach 2:
Text spans are pre-processed into embeddings and attention vectors before being used to generate the final structured representation. This preliminary action of creating intermediate representations (embeddings and attention vectors) prepares the data in a way that makes semantic relationships more detectable and easier to process, improving both reliability and ease of operation
Data Source
AI summary
Various embodiments of the present disclosure provide machine learning architectures and data processing techniques for improving computer-based text comprehension. The techniques may include identifying a plurality of data entity tokens from a target section of a multi-section natural language document and generating, using an embedding layer of a semantic chunking model, a text span embedding for a text span of the target section. The techniques may include leveraging the semantic chunking model to generate an attended span representation for the text span based on the text span embedding and the plurality of data entity tokens. The techniques may include identifying an entity topic that corresponds to the text span based on the attended span representation and, responsive to an identification of the entity topic, generating a subgraph data object for a knowledge graph using the text span.