Tabular Data Question Answering Using Column Row Correlation Classifiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods face challenges in efficiently identifying and extracting answers from large datasets stored in tables to answer user questions, particularly for lookup questions, as they require interpretation and processing of tabular data.
Innovation Solution
A computer-implemented method that uses classifiers to determine the probability of column and row correlations within a data table, employing pre-trained transformer models and linear regression models for next sentence prediction tasks, to suggest the most relevant cell containing the target answer, leveraging weak supervision and fine-tuning with training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional methods are used to search and interpret tabular data to answer questions, then the system can provide answers to user questions, but the process becomes inefficient and time-consuming when dealing with large datasets
Solution Approach 1:
The system performs preliminary classification by generating column and row correlation values before actually identifying the target cell. The classifiers are pre-trained and fine-tuned to evaluate correlations, so when a question is posed, the system can quickly compute which column and row are most likely to contain the answer without manually searching through the entire table. This preliminary evaluation of correlations speeds up the answer-finding process.
2Measurement precision
If the system processes and interprets tabular data to answer lookup questions, then accurate answers can be provided, but the complexity of the system increases
Solution Approach 1:
The system segments the complex task of identifying the target cell into two independent sub-tasks: (1) identifying the target column using a column classifier that computes column correlation values, and (2) identifying the target row using a row classifier that computes row correlation values. Each classifier operates independently on its respective dimension, making the overall system more manageable and interpretable while maintaining accuracy.
Solution Approach 2:
The system introduces correlation values as an intermediary metric between the question and the target cell identification. Instead of directly matching questions to cells, the system first computes how correlated each column and row is with the question, then uses these intermediate correlation scores to identify the most likely target cell. This intermediary step simplifies the overall matching process.
Data Source
AI summary
A computer answers a question using a data table. The computer receives a user question and a target table containing a target cell corresponding to a target answer for the user question, with the target cell corresponding to a target column and a target row. The computer generates, a first classifier to provide column correlation values reflecting the probability that a given column is the target column. The computer generates a second classifier that provides row correlation values reflecting the probability that a given row is the target row. The computer applies the first classifier to the target table to determine a column correlation value for each column. The computer applies the second classifier to the target table to determine a row correlation value for each row. The computer suggests, as the target cell, a cell having elevated column and row correlation values relative to other target table cells.