Document-Scale Embeddings for Structured Section Relationship Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative language models struggle to accurately capture relationships between different sections of structured documents, leading to misrepresentations and mis-training due to irrelevant information, which affects their relevance and accuracy in text generation.
Innovation Solution
The use of document embeddings that incorporate information about specific relationships between different segments of structured documents, generated by encoding segments with an encoder model and updating association vectors based on distance thresholds, to refine text model training and selection of seed tokens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative language models are trained on structured documents, then text generation capability is improved, but accuracy deteriorates due to inability to focus on relevant sections and ignore irrelevant information
Solution Approach 1:
The patent segments structured documents into multiple sections with hierarchical organization. Each section is processed separately to identify relevant information, allowing the model to focus on specific portions rather than treating the entire document as a single unit. This segmentation enables precise control over which information is attended to during text generation.
Solution Approach 2:
The patent applies local quality by creating section-specific embeddings that capture the unique characteristics and relevance of each document section. Different sections are encoded with different attention weights and embedding transformations, allowing the model to process each section with appropriate quality and focus rather than applying uniform processing to all content.
2Productivity
If document embeddings are generated without considering segment relationships, then processing speed is improved, but information completeness deteriorates due to missed associations between sections
Solution Approach 1:
The patent introduces a hierarchical dimension to document embeddings by organizing sections at multiple levels (document-level, section-level, and segment-level embeddings). This multi-dimensional embedding structure allows the model to capture relationships between sections while maintaining efficient processing through hierarchical aggregation rather than exhaustive pairwise comparisons.
Solution Approach 2:
The patent performs preliminary encoding of individual sections into embeddings before combining them into document-level representations. This preliminary action allows relationship information to be captured in advance through section-specific processing, enabling faster overall processing while preserving association information that would be lost in flat processing approaches.
3Device complexity
If all sections of structured documents are processed equally, then simplicity is maintained, but relevance accuracy deteriorates due to inability to distinguish important from irrelevant information
Solution Approach 1:
The patent implements dynamic processing by adjusting attention weights and embedding transformations based on section characteristics and relevance indicators. The model dynamically determines which sections require deeper processing and which can be processed more lightly, allowing simple yet adaptive processing that maintains accuracy without requiring uniformly complex processing of all sections.
Data Source
AI summary
A method and related system for generating document embeddings within an embedding space based on a set of structured documents by determining (i) a first vector based on a first segment of a first document and (ii) a second vector based on a second segment of the first document and updating association vectors indicating the second segment based on a distance between the first and second vectors. The method also includes generating a document embedding based on the association vectors, generating a candidate vector based on a candidate document, and determining a result indicating that a second distance between the candidate vector and a first document embedding satisfies a document embedding distance threshold. The method may also include generating a new document by providing, to a text generation model, a portion of the candidate document and a portion of the second segment of the first document.