Multimodal Table Encoding for Information Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval systems face challenges in efficiently indexing and searching tabular data within electronic documents, as they often return entire tables instead of relevant portions, leading to irrelevant results and reduced precision in responding to factual queries.
Innovation Solution
The method involves using separate machine learning encoders to encode the description, schema, rows, and columns of tables, along with end-of-column and end-of-row tokens, and applying a machine learning gating mechanism to produce a fused encoding that represents both the structure and content of the table, allowing for selective retrieval of relevant table portions in response to search queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate machine learning encoders are used to encode description, schema, rows, and columns separately, then the precision of table retrieval is improved, but the device complexity increases
Solution Approach 1:
The patent divides the table encoding process into multiple independent encoders: a description encoder for textual descriptions, a schema encoder for table structure, a row encoder for individual rows, and a column encoder for individual columns. Each encoder processes its specific modality separately and produces independent embeddings, which are then fused to create a comprehensive table representation. This segmentation allows each encoder to specialize in its modality, improving retrieval precision while maintaining modular system architecture.
2Measurement precision
If a fused encoding representing both structure and content is produced, then the relevancy of search results is improved, but the processing time increases
Solution Approach 1:
The patent pre-computes and stores fused embeddings for tables during an indexing phase, combining descriptions, schemas, rows, and columns into a single comprehensive representation that is saved for later retrieval. When a query is executed, the system retrieves these pre-computed embeddings and compares them against query embeddings, avoiding the need to perform complex fusion operations in real-time. This preliminary encoding action significantly reduces query processing time while maintaining high result relevancy.
3Manufacturing precision
If end-of-column and end-of-row tokens are included in encoding, then the structural accuracy is improved, but the quantity of data to be processed increases
Solution Approach 1:
The patent introduces special token embeddings as intermediary elements that mark structural boundaries within table data. End-of-column tokens are appended to column embeddings, and end-of-row tokens are appended to row embeddings. These token embeddings serve as mediators that encode structural information (column boundaries, row boundaries) without requiring complex structural annotations. The tokens act as simple yet effective markers that preserve table structure in the embedding space while adding minimal overhead to the data volume.
Data Source
AI summary
Multimodal table encoding, including: Receiving an electronic document that contains a table. The table includes multiple rows, multiple columns, and a schema comprising column labels or row labels. The electronic document includes a description of the table which is located externally to the table. Next, operating separate machine learning encoders to separately encode the description, schema, each of the rows, and each of the columns of the table, respectively. The schema, the rows, and the columns are encoded together with end-of-column tokens and end-of-row tokens that mark an end of each column and row, respectively. Then, applying a machine learning gating mechanism to the encoded description, encoded schema, encoded rows, and encoded columns, to produce a fused encoding of the table, wherein the fused encoding is representative of both a structure of the table and a content of the table.