Hierarchical Transformer Encoding for Long-Document Semantic Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional models struggle with efficient semantic matching between long documents due to quadratic computational complexity and inability to understand complex internal structures, limiting their application to short textual segments.
Innovation Solution
A machine-learned semantic document encoding model with a hierarchical siamese transformer network structure processes documents in a hierarchical manner, reducing complexity by localizing dependencies between textual segments and modeling long-distance dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional semantic analysis models are used to match long documents, then semantic understanding capability is achieved, but computational cost grows quadratically with document length
Solution Approach 1:
The patent segments long documents into multiple textual blocks (e.g., paragraphs or sections) and processes each block independently through the transformer encoder. This divides the original quadratic complexity problem into multiple smaller sub-problems, where only local attention computations are performed within each block rather than global attention across all tokens in the entire document.
Solution Approach 2:
The patent introduces a hierarchical dimension by organizing document processing into two levels: block-level representations and document-level representations. The block encoder operates at the local level within each textual block, while the document encoder operates at the global level across all blocks. This dimensional hierarchy transforms the single-layer quadratic complexity into a multi-layer structure with reduced overall computational burden.
2Measurement precision
If conventional semantic analysis models process long documents, then complete document understanding is achieved, but processing time increases significantly
Solution Approach 1:
By segmenting the document into textual blocks and processing them in parallel through the block encoder, the patent reduces sequential processing time. Each block can be encoded independently and simultaneously, eliminating the need to process tokens sequentially across the entire document length.
Solution Approach 2:
The block encoder performs preliminary encoding of individual textual blocks into block-level representations before the document encoder performs final document-level encoding. This preliminary action at the block level prepares condensed representations that capture local semantic information, reducing the processing burden on the subsequent document-level encoding stage.
3Measurement precision
If conventional models analyze complex document structures with multiple textual segments, then comprehensive semantic analysis is achieved, but model complexity increases
Solution Approach 1:
The patent segments the monolithic transformer model into two distinct encoder components: a block encoder for local textual block processing and a document encoder for global document-level processing. This segmentation allows each component to be optimized for its specific function, reducing overall model complexity while maintaining comprehensive semantic analysis capability.
Solution Approach 2:
The patent introduces a hierarchical dimension with block-level and document-level encoding layers. The block encoder handles local semantic relationships within textual blocks, while the document encoder handles global semantic relationships across blocks. This dimensional separation simplifies the model architecture by distributing complexity across hierarchical levels rather than concentrating it in a single layer.
4Measurement precision
If transformer networks perform attention computation across all document tokens, then accurate semantic matching is achieved, but computational resources required increase quadratically
Solution Approach 1:
The patent segments the attention computation scope from global (across all document tokens) to local (within each textual block). The block encoder performs attention computation only on tokens within the same block, dramatically reducing the quadratic complexity from O(N²) for the entire document to O(n²) for each small block, where n is the block size.
Solution Approach 2:
The patent adds a hierarchical dimension to attention computation by implementing block-level attention within the block encoder and document-level attention across block representations in the document encoder. This dimensional hierarchy allows the system to capture both local and global semantic relationships while keeping computational resources manageable at each level.
Data Source
AI summary
Systems and methods of the present disclosure are directed to a method for predicting semantic similarity between documents. The method can include obtaining a first document and a second document. The method can include parsing the first document into a plurality of first textual blocks and the second document into a plurality of second textual blocks. The method can include processing each of the plurality of first textual blocks and the second textual blocks with a machine-learned semantic document encoding model to obtain a first document encoding and a second document encoding. The method can include determining a similarity metric descriptive of a semantic similarity between the first document and the second document based on the first document encoding and the second document encoding.