Semantic Feature Derivation for ML Data Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data science methods rely heavily on human intervention for feature derivation and processing, which is labor-intensive and inefficient, especially when dealing with large datasets or datasets with many features, leading to suboptimal feature usage in machine learning algorithms.
Innovation Solution
The implementation of a system that uses semantic processing to automatically derive additional features from existing features in a dataset based on semantic context, allowing for the identification and tagging of feature types, relationships, and the application of algorithms to generate new features with varying semantic contexts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual feature derivation by data scientists is used, then feature quality and relevance can be ensured, but the process becomes labor-intensive and time-consuming
Solution Approach 1:
The system performs preliminary action by automatically generating multiple candidate features from existing features before the data scientist needs to select the best ones. The semantic processing pre-computes potential features that would otherwise require manual derivation, allowing data scientists to review and select from pre-generated options rather than creating features from scratch.
Solution Approach 2:
The system enables self-service by allowing the data processing system to automatically derive features from existing features without requiring constant human intervention. The semantic processing algorithms independently analyze existing features and generate new features based on semantic relationships, reducing the manual labor required while maintaining feature quality through automated semantic analysis.
2Measurement precision
If more features are used in machine learning algorithms, then model accuracy can be improved, but machine learning algorithms do not scale well with large numbers of features
Solution Approach 1:
The system applies extraction by identifying and extracting only the most relevant features from a large set of semantically-derived candidate features. Rather than using all generated features, the system extracts the subset that provides the most value for machine learning, reducing complexity while maintaining accuracy benefits from having access to many potential features.
Solution Approach 2:
The system uses partial action by generating more features than will ultimately be used, then selecting the optimal subset. This excessive generation of candidate features ensures that the best features are available, but only the necessary portion is fed into machine learning algorithms, balancing comprehensiveness with computational scalability.
3Productivity
If automated semantic processing is implemented, then processing efficiency increases, but the system complexity increases
Solution Approach 1:
The system achieves universality by implementing a multi-functional semantic processing layer that can handle various feature derivation tasks through a single unified framework. This semantic processing system serves multiple purposes: generating new features, validating existing features, and providing semantic context for machine learning, thereby justifying the added complexity through multiple benefits.
Solution Approach 2:
The semantic processing system acts as an intermediary layer between raw data and machine learning algorithms. This mediator translates and enriches existing features with semantic context, enabling automated feature derivation without requiring complex modifications to the underlying machine learning algorithms themselves. The intermediary handles the complexity internally while presenting simplified interfaces to users.
4Reliability
If human data scientists manually clean and normalize datasets, then data quality can be ensured, but the process is very labor intensive
Solution Approach 1:
The system implements self-service by automatically performing data cleaning and normalization tasks that previously required manual human intervention. The semantic processing algorithms independently identify and correct data quality issues, normalize features according to semantic type, and validate data consistency without requiring data scientists to manually process each dataset, thereby maintaining quality while increasing throughput.
Data Source
AI summary
Techniques for deriving additional features from input data are described herein. Input data from a plurality of source files are received. One or more features corresponding to the input data, which includes information about semantic types, is identified. The input data is then processed to generate additional features for the input data. New data corresponding to the additional features are then generated and access to the new data is subsequently provided.