Structure-Aware Sequence Model for Non-Standard Document Layouts

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Natural language processing models face challenges in interpreting text from documents with non-standard layouts, such as marketing documents, menus, and photographs, due to difficulties in serializing text correctly, which affects their ability to draw meaningful information.

Innovation Solution

The implementation of a structure-aware sequence model using a graph convolutional network (GCN) and a transformer with a sparse attention paradigm, which generates contextualized supertoken embeddings and adjusts attention scores based on predicted and actual order and distance between tokens, preserving local syntactic information and reducing the need for correct reading layout inference.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a language model uses traditional sequence modeling approaches to process text, then it can handle simple documents with standard layouts, but it fails to accurately process documents with non-standard layouts (marketing documents, menus, photographs, columns, tables, text around pictures)

Engineering Contradiction:

Improveability to process documents with nonstandard layoutsVSAvoidaccuracy of text serialization

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent segments text processing into two distinct phases: (1) layout analysis phase that identifies spatial relationships and reading order using OCR and geometric calculations, and (2) NLP phase that processes the serialized text. This segmentation allows the model to handle non-standard layouts by first understanding their structure before applying language modeling, thereby improving both adaptability to different layouts and reliability of text processing.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary layout analysis component that acts as a bridge between the input document image and the NLP model. This intermediary extracts spatial information, determines reading order, and serializes text accordingly before passing it to the language model. This mediator enables the system to adapt to various document layouts while maintaining reliable text processing through the NLP model.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If a language model is trained to infer reading order from documents, then it can handle various layouts, but the model size and training requirements increase significantly

Engineering Contradiction:

Improveability to infer reading layoutVSAvoidmodel size and training requirements

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent extracts the layout analysis function from the main language model and implements it as a separate, lightweight preprocessing component. Instead of training the large NLP model to infer reading order, the system extracts spatial information using simpler OCR and geometric algorithms, then feeds the serialized text to the language model. This extraction reduces model size and training requirements while maintaining adaptability to various layouts.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent replaces the complex neural network-based reading order inference mechanism with a deterministic geometric calculation system. By using coordinate-based spatial relationships and predefined geometric rules to determine reading order, the system eliminates the need for the language model to learn layout patterns, significantly reducing model complexity and training requirements while preserving layout adaptability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Loss of information

If text is properly serialized from documents with nonstandard layouts, then the language model can understand the text, but the serialization process becomes more complex and error-prone

Engineering Contradiction:

Improvepreservation of text meaningVSAvoidcomplexity of serialization process

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent performs preliminary layout analysis and text serialization before the main NLP processing. By pre-processing the document to extract text in the correct reading order based on spatial relationships, the system eliminates the need for the language model to handle layout complexity during training or inference. This preliminary action preserves text meaning while simplifying the overall processing pipeline.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces complex heuristic-based text serialization with a deterministic geometric calculation system. By using coordinate-based spatial relationships and mathematical rules to determine text order, the system achieves accurate serialization without requiring complex machine learning models, thereby reducing errors while maintaining preservation of text meaning.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240354504A1Structural encoding and attention paradigms for sequence modeling

Publication Date: 2024.10.24 GOOGLE LLC

AI summary

Systems and methods for providing a structure-aware sequence model that can interpret a document's text without first inferring the proper reading order of the document. In some examples, the model may use a graph convolutional network to generate contextualized “supertoken” embeddings for each token, which are then fed to a transformer that employs a sparse attention paradigm in which attention weights for at least some supertokens are modified based on differences between predicted and actual values of the order and distance between the attender and attendee supertokens.