ML-Based Preprocessing Identification for Dataset Feature Changes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing preprocessing identification methods rely on documents, which limits accuracy and effectiveness, as they do not directly reflect preprocessing contents and require a corresponding document to function effectively.
Innovation Solution
An information processing device that uses a trained machine learning model to identify similar preprocessing by analyzing feature changes in datasets before and after preprocessing, generating meta-features, and associating them with preprocessing information, allowing for the selection of appropriate preprocessing without relying on documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If preprocessing identification methods rely on documents, then the system can function with available documentation, but the accuracy and effectiveness of preprocessing identification deteriorates because documents do not directly reflect preprocessing contents
Solution Approach 1:
The patent replaces the document-based identification system with a machine learning model-based system. Instead of relying on textual documents that indirectly describe preprocessing, the system uses a trained machine learning model that directly processes dataset features and metadata to identify appropriate preprocessing operations, thereby substituting an indirect information source with a direct computational approach
Solution Approach 2:
The patent introduces a machine learning model as an intermediary between the dataset and the preprocessing identification process. The model takes dataset features and metadata as input and outputs preprocessing recommendations, serving as a mediator that translates raw data characteristics into actionable preprocessing decisions without requiring intermediate documentation
2Adaptability or versatility
If preprocessing identification requires corresponding documents to function effectively, then document-based methods can be used, but the adaptability and applicability deteriorates when documents are unavailable
Solution Approach 1:
The patent enables the preprocessing identification system to be self-sufficient by training the machine learning model on dataset characteristics and metadata. The system serves itself by learning from the data without requiring external documentation, allowing it to function autonomously and adapt to different datasets based on their inherent features rather than relying on pre-written guides
Solution Approach 2:
The patent changes the input parameters from document text to dataset features and metadata. By transforming the identification basis from textual documentation to numerical and structural data characteristics, the system becomes adaptable to various datasets regardless of documentation availability, using the actual data parameters rather than descriptions of them
3Productivity
If the search space for preprocessing is not narrowed, then all preprocessing options can be considered, but the complexity and time required for preprocessing selection increases
Solution Approach 1:
The patent segments the preprocessing selection process by categorizing preprocessing operations into distinct types (e.g., normalization, encoding, transformation) and training separate machine learning models or using hierarchical filtering for each category. This segmentation allows the system to evaluate only relevant preprocessing types for each dataset rather than considering all possible preprocessing operations, reducing search complexity while maintaining comprehensiveness
Data Source
AI summary
A non-transitory computer-readable recording medium stores a program for causing a computer to execute a process, the process includes obtaining first change information, which indicates a change in a feature of a first dataset when first preprocessing is performed on the first dataset, inputting the first change information to a trained machine learning model that outputs an inference result regarding preprocessing information that identifies each piece of second preprocessing for a second dataset, the trained machine learning model being trained by using training data in which the preprocessing information is associated with second change information that indicates a change in a feature of the second dataset when each piece of second preprocessing is performed, and identifying one or more pieces of recommended preprocessing that correspond to the first preprocessing based on the inference result that is output in response to the input of the first change information.