Document Relevance Modeling With Section Embeddings and GCNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods struggle to determine the relevance of input files to predefined topics when the files have variable lengths or non-standard formats, leading to inefficiencies and high costs in using transformer models, especially when dealing with long documents.
Innovation Solution
A computer-implemented method using a graph convolutional network (GCN) that processes embeddings generated from sections of input files, constructed as a graph, to determine relevance, combining pre-trained and end-to-end trained transformer models to handle variable document lengths efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If transformer models are used to process documents of variable lengths, then relevance determination accuracy is improved, but computational cost and processing time increase significantly
Solution Approach 1:
The patent divides long documents into multiple sections or chunks, processes each section separately through the transformer model, and then aggregates the results. This segmentation approach maintains accuracy by ensuring each section fits within the model's context window while significantly reducing the computational burden compared to processing the entire document at once.
2Productivity
If documents are padded to standard length for transformer model input, then model processing efficiency is improved, but information loss and irrelevance increase
Solution Approach 1:
Instead of padding documents to a fixed length, the patent segments documents into meaningful sections that fit within the model's context window. This approach maintains processing efficiency by providing standardized input sizes while avoiding the information loss and irrelevance introduced by padding with dummy tokens.
3Productivity
If subdocument level predictions with heuristic rules are used, then computational efficiency is improved, but relevance determination accuracy deteriorates
Solution Approach 1:
The patent segments documents into sections and processes each section through a transformer model to generate embeddings, then uses a graph convolutional network to aggregate these embeddings and determine overall document relevance. This approach improves upon simple heuristic aggregation by using learned representations and graph-based aggregation to capture relationships between sections, thereby maintaining computational efficiency while significantly improving accuracy.
4Adaptability or versatility
If long context models are used to handle variable length documents, then document length flexibility is improved, but model size and training cost increase prohibitively
Solution Approach 1:
The patent avoids using prohibitively large long context models by segmenting documents into manageable sections that fit within standard model context windows. This segmentation strategy maintains document length flexibility while using computationally efficient models, eliminating the need for excessively large and expensive long context models.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a computer-implemented method of determining a relevance of an input file to a predefined topic. The computer-implemented method comprises: receiving (S100) an input file; dividing (S102) the input file into a plurality of sections; inputting (S104) each section of the plurality of sections into at least one transformer model to generate at least one embedding for each section; constructing (S106) a graph using the at least one embedding for each section; inputting (S108) the graph to a graph convolutional network, GCN, to generate a relevance of the input file to a predefined topic; and outputting (S110) the relevance. Also provided is a method of training a machine learning model including the first and second transformers, and the GCN.