Table Retrieval Using Intrinsic and Extrinsic Similarity Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information retrieval systems face challenges in effectively ranking relevant tables from large datasets, as they often fail to accurately capture the relevance of tabular data due to the lack of focused retrieval approaches and proper utilization of table modalities.
Innovation Solution
The method represents tables as multi-field text documents, combining different modalities such as titles, captions, schemas, and tabular data into separate fields, and calculates intrinsic and extrinsic similarity scores using passage-based and manifold-based ranking techniques to enhance retrieval relevancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If tables are represented as single-field documents, then the retrieval process is simpler, but the retrieval precision and relevance are lower
Solution Approach 1:
The patent divides table representation into multiple fields including title, caption, schema, and data fields. Each field captures different aspects of the table's meaning and structure, allowing the retrieval system to search and rank tables based on multiple relevant features simultaneously, thereby improving retrieval precision while maintaining systematic processing
2Productivity
If only intrinsic similarity is used for ranking, then the computation is faster, but the ranking accuracy is lower
Solution Approach 1:
The patent performs preliminary retrieval using intrinsic similarity scores computed from the multi-field table representations. This initial filtering step quickly identifies candidate tables that are potentially relevant, and then extrinsic similarity computation is applied only to these candidates for final re-ranking, balancing computational efficiency with ranking accuracy
3Measurement precision
If only extrinsic similarity is used for ranking, then the ranking is more accurate, but the computation time increases significantly
Solution Approach 1:
The patent computes extrinsic similarity scores and uses them to re-rank only the candidate tables that have already been identified through preliminary retrieval using intrinsic similarity. This two-stage approach ensures that the computationally intensive extrinsic similarity calculation is performed only on a small subset of candidate tables, maintaining both accuracy and efficiency
4Device complexity
If all table modalities are combined into a single field, then the representation is simpler, but the retrieval relevancy is lower
Solution Approach 1:
The patent segments table modalities into distinct fields (title, caption, schema, data) that can be independently processed and weighted. This allows the retrieval system to give different importance to different aspects of the table based on the query, improving retrieval relevancy while maintaining a structured and manageable representation format
Data Source
AI summary
Ad-hoc table retrieval, including: Representing each of a plurality of tables as a multi-field text document in which: different modalities of the table are represented as separate fields, and a concatenation of all the modalities is represented as a separate, auxiliary field. Receiving a query. Executing the query on the multi-field text documents, to retrieve a list of preliminarily-ranked candidate tables out of the plurality of tables. Calculating an intrinsic table similarity score for each of the candidate tables, based on the query and the auxiliary field. Calculating an extrinsic table similarity score for each of the candidate tables, based on a cluster hypothesis of the candidate tables. Combining: the preliminary rankings, the intrinsic table similarity scores, and the extrinsic table similarity scores, to re-rank the candidate tables.