Automated Imputation System for ML Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for imputing missing data in machine learning datasets are inefficient, as they often require manual selection of algorithms, are dependent on memory-based solutions, and struggle with non-MCAR or non-MAR scenarios, making them unsuitable for large datasets and real-world applications.
Innovation Solution
An automated system that analyzes the dataset to identify characteristics of missing values, selects the optimal imputation algorithm, and determines the appropriate computing environment for execution, generating code that can be integrated into a machine learning pipeline for efficient imputation of missing values, including non-memory-based solutions for large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual selection of imputation algorithms is used, then flexibility in choosing appropriate methods is improved, but time consumption and operational complexity increase
Solution Approach 1:
The system performs self-service by automatically analyzing dataset characteristics and selecting appropriate imputation algorithms without manual intervention. The automated system evaluates data types, missing value patterns, and dataset properties to autonomously determine the most suitable imputation method, eliminating the time-consuming manual selection process while maintaining adaptability.
Solution Approach 2:
The system changes parameters by automatically adjusting algorithm selection based on dataset characteristics such as data type (categorical/numerical), missing value percentage, and distribution patterns. This dynamic parameter adjustment enables the system to adapt to different scenarios without manual reconfiguration, resolving the contradiction between flexibility and time consumption.
2Ease of manufacture
If memory-based solutions are used for imputation, then ease of implementation is improved, but applicability to large datasets deteriorates
Solution Approach 1:
The system segments the imputation process into multiple stages: data analysis, algorithm selection, and execution. For large datasets, it further segments the data processing by utilizing distributed computing frameworks and cloud-based services, breaking down the imputation task into manageable chunks that can be processed in parallel, thus maintaining ease of implementation while enabling scalability to large datasets.
Solution Approach 2:
The system introduces intermediary components such as cloud-based computing services and distributed processing frameworks that mediate between the imputation algorithm and large datasets. These intermediaries enable the system to handle large-scale data without requiring all data to reside in memory simultaneously, resolving the contradiction between implementation ease and large dataset applicability.
3Productivity
If simple imputation methods are used, then computational efficiency is improved, but model accuracy deteriorates
Solution Approach 1:
The system dynamically selects imputation methods based on dataset characteristics rather than using a fixed simple method. For datasets where simple imputation is appropriate, it uses efficient methods like mean/median imputation. For complex datasets with specific patterns (e.g., missing not at random, correlated features), it automatically selects more sophisticated methods like iterative imputation or model-based approaches, thus balancing computational efficiency with model accuracy dynamically.
Solution Approach 2:
The system changes the complexity parameter of imputation methods based on dataset properties such as missing value percentage, data distribution, and feature correlations. This parameter adjustment allows the system to use simpler methods for straightforward cases (maintaining efficiency) and more complex methods for challenging cases (improving accuracy), resolving the contradiction between productivity and reliability.
4Ease of operation
If existing imputation packages are used for large datasets, then ease of use is improved, but memory requirements cause inapplicability
Solution Approach 1:
The system transitions from a single-dimension local memory-based approach to multi-dimensional solutions by incorporating cloud-based computing resources and distributed storage systems. This dimensional expansion allows the system to access virtually unlimited storage and computing power remotely, enabling easy operation on large datasets without being constrained by local memory space limitations.
Data Source
AI summary
Methods and apparatuses are described for intelligent imputation of missing data in a machine learning (ML) dataset comprised of a plurality of features. Each feature includes a plurality of values, where at least a portion of the values for one or more features are missing. A server analyzes the ML dataset to generate characteristics of the missing values in the ML dataset. The server selects an imputation algorithm for filling in the missing values based upon the identified characteristics. The server determines a computing environment in which the imputation algorithm is executed based upon one or more of a size of the ML dataset or the selected algorithm. The server generates code that comprises instructions for executing the imputation algorithm on the ML dataset in the computing environment. The server integrates the code into an ML platform that executes the code to assign replacement values to the missing values.