Hybrid Label Identification for Unlabelled Tabular Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data labelling techniques face challenges with unlabelled tabular data, particularly due to the lack of column information, requiring large amounts of sample data for training and failing to handle various data types like text, regex, and numerical data accurately.
Innovation Solution
A hybrid approach using language models, regular expressions, and predefined dictionaries for labelling unlabelled tabular data, employing intelligent bucketing and feedback mechanisms to ensure accurate label identification, with AI and domain knowledge refinement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If semi-supervised models are used for labelling unlabelled tabular data, then label identification can be performed, but large amounts of sample labelled data are required for training and the system fails when training data labels are erroneous
Solution Approach 1:
The patent segments the labelling system into multiple independent components: a labelling module that handles text data using NLP techniques, a numerical module for numerical data using statistical analysis, and a regex module for pattern matching. Each module operates independently on its designated data type, eliminating the need for large sample datasets for training while maintaining high accuracy for each specific data category.
2Adaptability or versatility
If a single labelling model is used for all data types, then the system is simple, but it cannot accurately handle text, regex, and numerical data types simultaneously
Solution Approach 1:
The patent creates a universal data processing platform that handles multiple data types through a common architecture. The system includes a unified data reception interface, a central processing orchestration layer, and standardized output formatting, while incorporating specialized sub-modules (NLP, numerical analysis, regex) that can process different data types. This multi-functional design enables the system to adapt to various data types without requiring separate independent systems for each type.
3Ease of manufacture
If provided labelled data contains discrepancies in labels, then data is available for training, but the training quality deteriorates and correct column data patterns cannot be ensured
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically validates and cleanses the training data before use. The labelling modules incorporate built-in validation rules and consistency checks that identify and correct erroneous labels without external intervention. The system performs automated data quality assessment, detects anomalies in provided labelled data, and applies correction algorithms to ensure training data meets quality standards before model training begins.
Data Source
AI summary
As discussed earlier, labelling techniques that are available for labelling of unlabelled tabular data use some semi supervised models for identification purposes. However, they require sample labeled data for training purposes. Further, the same labelling model/technique cannot be used for all data types. Present disclosure provides method and system for identifying labels of unlabeled column data. The system uses a hybrid approach i.e., it uses language models, regular expressions and known dictionaries for labelling of unlabelled tabular data. For performing labelling, system first classifies received unlabelled tabular data into one or more data buckets. The system then uses appropriate techniques, based on data types, for identification of labels of unlabeled data present in data buckets. Thereafter, system uses feedback mechanism which will impart maturity to system over time. Finally, once system is matured, system can identify labels for all types of data.