Transformer Clause Comparison Using Segmented Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language processing techniques, particularly transformer models, face limitations in comparing long texts due to quadratic complexity and attention mechanism constraints, which restrict input sequence lengths and accuracy in semantic textual similarity tasks, especially when dealing with arbitrarily long documents like contracts divided into clauses.
Innovation Solution
A transformer architecture is used for document ingestion, converting natural language text into embeddings and storing them in a database, allowing for comparison of query clauses with document clauses by generating query-conditioned embeddings and calculating similarity scores, enabling scalable and accurate comparison of clauses across varying lengths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer models with attention mechanism are used for semantic textual similarity, then accuracy in understanding contextual meaning is improved, but input sequence length is limited to 512 tokens due to quadratic complexity
Solution Approach 1:
The patent divides long documents into multiple clauses, each clause being processed independently by the transformer model. This segmentation allows the model to maintain the 512-token input limit while handling arbitrarily long documents by processing them in manageable chunks that preserve semantic meaning.
Solution Approach 2:
The patent introduces a new dimension of comparison by generating multiple candidate clauses from different documents and comparing them against a query clause. This multi-dimensional approach allows the system to handle long documents without requiring the entire document to fit within the 512-token window, as only relevant clauses are compared.
2Reliability
If transformer models process pairwise comparisons across billions of documents, then comprehensive text comparison is achieved, but time complexity becomes prohibitively expensive
Solution Approach 1:
By segmenting documents into clauses and only comparing relevant clauses rather than entire documents, the patent dramatically reduces the number of pairwise comparisons needed. This segmentation approach maintains comprehensive comparison coverage while reducing computation time from billions of document-level comparisons to a manageable number of clause-level comparisons.
Solution Approach 2:
The patent generates multiple candidate clauses (excessive action) to ensure comprehensive coverage, but only performs detailed similarity comparisons on the most relevant candidates rather than all possible pairs. This partial action approach balances thoroughness with computational efficiency.
3Length of moving object
If document-level transformer models aggregate input sentence embeddings to handle arbitrarily long documents, then scalability to long texts is improved, but information preservation deteriorates as the model forgets beginning information
Solution Approach 1:
The patent segments long documents into multiple clauses and processes each clause independently through the transformer model. This segmentation prevents information loss by ensuring that no clause exceeds the 512-token window, thereby preserving all information without requiring the model to retain memory of beginning information across entire long documents.
Solution Approach 2:
The patent performs preliminary processing by dividing documents into clauses before transformation, ensuring that each clause is within the optimal input window. This preliminary action prevents information loss by structuring the input data in a way that the transformer can process each segment with full contextual awareness.
Data Source
AI summary
System, method, apparatus, and program instruction for comparing an input query clause with a collection of document clauses to determine which document clause is most similar to the query clause is provided. The disclosed invention includes an improved process of storing a collection of natural language data, improving both the generalizability and accuracy of searching and comparison of natural language data.