Graphical Data Preparation Interface for ML Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data scientists face challenges in efficiently analyzing and preparing data for machine learning due to difficulties in connecting to various data sources and configuring tools for data cleansing, transformation, and feature engineering, making these steps non-automated and reproducible.
Innovation Solution
An interactive graphical user interface for machine learning data preparation environments allows data scientists to easily connect to data sources, leverage built-in data-science-specific transforms, and visually represent data preparation workflows as graphs, which are converted into executable instructions by a backend service for data cleansing and transformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If data scientists manually connect to data sources and configure transformation tools, then data preparation can be performed with flexibility, but the process becomes time-consuming and non-reproducible
Solution Approach 1:
The patent creates reproducible data preparation pipelines by copying and reusing configured transformation steps. The system allows data scientists to define transformation logic once and then replicate it across multiple datasets and workflows, ensuring consistency and eliminating manual reconfiguration while maintaining flexibility through parameterization.
Solution Approach 2:
The system segments the data preparation process into discrete, modular transformation steps that can be independently configured and reused. Each transformation step is separated into distinct operations (connection, transformation, validation) that can be assembled into different pipelines, reducing overall complexity while enabling automation through standardized interfaces.
2Productivity
If data scientists perform manual data cleansing and transformation, then they can handle complex data scenarios, but productivity decreases due to manual effort
Solution Approach 1:
The system enables self-service data preparation by providing automated transformation steps that execute without manual intervention. The pipeline framework automatically connects to data sources, applies configured transformations, and validates outputs based on predefined parameters, allowing data scientists to focus on high-value analysis rather than repetitive manual tasks.
Solution Approach 2:
The patent implements preliminary action by pre-configuring transformation steps and validation rules before actual data processing occurs. The system allows data scientists to define and test transformation logic in advance using sample data, then reuse these pre-configured steps on production datasets, significantly reducing the time required for repetitive data preparation tasks.
3Reliability
If data preparation steps are not standardized, then adaptability to different data sources is maintained, but reproducibility and reliability are compromised
Solution Approach 1:
The system maintains adaptability through parameterization, allowing the same standardized pipeline template to work with different data sources by changing connection parameters and transformation parameters. The framework uses configurable parameters (connection strings, transformation types, validation thresholds) that can be adjusted for each dataset while maintaining the same reproducible execution logic, thus achieving both reliability and versatility.
Data Source
AI summary
Systems and methods to obtain a text-based representation of a machine learning (ML) graph identifying one or more transforms usable to prepare data for ML training. The systems and methods can determine computer-executable instructions based on the text-based representation of the ML graph, where the computer-executable instructions can include instructions associated with the one or more transforms to prepare data for ML training. Additionally, the systems and methods can process the computer-executable instructions to generate ML training data based on at least the one or more transforms.