Tabular Foundation Model Training via Feature Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current foundation models struggle to accurately find meaningful data within tabular data sets, as they are primarily trained on unstructured data and lack the ability to effectively handle tabular data structures.
Innovation Solution
A computer-implemented method that involves receiving tabular data records, generating clusters associated with real-world entities, identifying informative features, masking a subset of these features, and training a tabular foundation model using self-supervision techniques based on the masked features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If foundation models are trained on unstructured data, then they can understand natural language and generate content, but they struggle to accurately find meaningful data within tabular data sets
Solution Approach 1:
The patent segments the training process into distinct phases: pre-training on unstructured data to build general capabilities, then fine-tuning on tabular data with specific techniques (masking, clustering, entity matching) to develop tabular-specific skills. This segmentation allows the model to maintain versatility while achieving precision for tabular data tasks.
Solution Approach 2:
The patent applies preliminary actions by pre-training the foundation model on unstructured data before adapting it to tabular data. Additionally, during tabular training, informative features are identified and masked in advance to prepare the model for learning meaningful patterns. Entity matching and clustering are performed preliminarily to organize data before the actual training occurs.
2Productivity
If foundation models are trained on large-scale unstructured data, then they can generate new content and understand language, but they lack the ability to effectively handle tabular data structures
Solution Approach 1:
The patent makes the foundation model universal by enabling it to perform both unstructured data tasks (language understanding, content generation) and tabular data tasks (entity matching, data imputation, error detection). The model maintains its original capabilities while acquiring new tabular-specific skills through the proposed training methodology, achieving multi-functionality across different data types.
Solution Approach 2:
The patent changes training parameters when transitioning from unstructured to tabular data. Specifically, it modifies the training objective to include masking informative features, adjusts the data preprocessing to create clusters and identify entities, and modifies loss functions to account for tabular data characteristics. These parameter changes enable the model to adapt to tabular structures while preserving general capabilities.
3Measurement precision
If self-supervision techniques are used to train on masked features, then the model can learn meaningful patterns in tabular data, but the training process becomes more complex
Solution Approach 1:
The patent implements self-service through self-supervised learning where the model learns to predict masked informative features without requiring external supervision or manual labeling. The system automatically generates training signals by masking features and using the model's own predictions as learning targets. Entity matching and clustering provide self-generated supervision signals, reducing the need for complex external annotation processes while improving learning accuracy.
Data Source
AI summary
A processor set is configured to receive tabular data records and generate a plurality of clusters, associated with specific real-world entities, within the received tabular data records, wherein each cluster is associated with a specific real-world entity. The processor set may further identify informative features within a first cluster and mask a subset of the informative features. Based on the masked subset of informative features and using self-supervision techniques, the processor set may train a tabular foundation model.