ML Header and Column Models for Tabular Data Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database extract, transform, and load processes struggle with automated recognition and labeling of data from input sources, particularly in tabular data with unclear header rows and columns.
Innovation Solution
A computer-implemented method using machine learning models to identify and label data in tabular formats. This involves training a machine learning header model to identify header rows and a machine learning column model to apply defined labels to columns, based on input data features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are used to automatically identify header rows and label columns, then the accuracy and efficiency of data processing is improved, but the complexity of the system increases
Solution Approach 1:
The system divides the data processing task into separate machine learning models: a header row identification model and multiple column-specific labeling models. Each model handles a specific function, allowing the system to achieve high accuracy through specialized processing while managing complexity through modular architecture.
Solution Approach 2:
The system introduces an intermediary layer of machine learning models between the raw input data and the final labeled output. These models act as mediators that automatically interpret and label data without human intervention, improving accuracy while the models themselves encapsulate the complexity.
2Measurement precision
If multiple column-specific machine learning models are used to determine column labels, then the precision of column labeling is improved, but the processing time and computational resources increase
Solution Approach 1:
The system segments the labeling task into separate column-specific models, each trained to recognize specific data types or patterns. This allows parallel processing of different columns simultaneously, maintaining high precision while reducing overall processing time through concurrent execution.
Solution Approach 2:
The system applies machine learning models selectively to columns based on their characteristics. Not all columns require the same level of analysis, so the system uses partial action by applying appropriate models only where needed, optimizing the balance between precision and processing time.
Data Source
AI summary
A computer-implemented method includes receiving input data that is organized into a set of rows and a set of columns. The method includes maintaining a machine learning header model that is trained on tabular data with header rows. The method includes supplying the input data as input to the machine learning header model to generate header row identification data that identifies a set of header rows that is a subset of the set of rows. The method includes maintaining a machine learning column model that is trained on tabular data. The method includes supplying the header row identification data and features of the input data to the machine learning column model to generate column label data that applies a set of defined labels to the set of columns. The method includes generating output data that is organized into rows and columns.