Transformer Model for Tabular Data Discovery Using Sketches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently navigating and discovering relevant tabular data within data lakes, particularly in identifying unionable, joinable, and subset tables, due to the complexity and volume of data.
Innovation Solution
A computer-implemented method using a transformer architecture machine learning model that combines sketches of tabular data columns with metadata embeddings to form input vectors, enabling improved dataset discovery and navigation within data lakes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data navigation methods are used in data lakes, then basic data retrieval is possible, but the ability to efficiently discover relevant tabular data (unionable, joinable, subset tables) deteriorates due to data complexity and volume
Solution Approach 1:
The patent introduces sketches as intermediary structures that summarize tabular data characteristics without requiring full data inspection. These sketches serve as mediators between the complex data lake and the discovery system, enabling efficient comparison and matching of tables based on their statistical properties rather than raw data content.
Solution Approach 2:
The system transforms tabular data into parameter-based representations through sketches that capture statistical properties (mean, variance, cardinality, etc.). This parameter transformation allows the discovery system to work with compact numerical representations instead of complex tabular structures, dramatically improving discovery efficiency while handling data complexity.
2Measurement precision
If comprehensive data analysis is performed to improve discovery accuracy, then identification of relevant tables improves, but computational time and resources worsen
Solution Approach 1:
The patent applies partial action by computing only essential statistical parameters (sketches) rather than performing complete data analysis. This selective computation captures the most relevant characteristics for discovery tasks while avoiding unnecessary computational overhead, achieving good discovery accuracy with reduced time investment.
Solution Approach 2:
The system performs preliminary computation of sketches that summarize key data characteristics before the actual discovery process. This preliminary action prepares compact representations that enable fast subsequent comparisons and matching operations, reducing the computational time required for comprehensive data analysis during discovery.
3Loss of information
If detailed metadata is used to represent tabular data, then data characterization improves, but the complexity of processing and storing metadata worsens
Solution Approach 1:
The patent extracts only the most essential characteristics from tabular data to create sketches, omitting redundant or less useful metadata. This selective extraction maintains adequate data characterization for discovery tasks while significantly reducing metadata volume and processing complexity compared to using complete data descriptions.
Data Source
AI summary
A method, computer system, and a computer program product are provided. Training data in tabular form having at least some columns is received. One or more sketches for contents of the respective columns are created. The sketches are combined with metadata embeddings of the training data to form respective combined input vectors. A transformer architecture machine learning model is trained by computing loss based on an objective function, by inputting the combined input vectors into the transformer architecture machine learning model, and, in response, the transformer architecture machine learning model producing an output.