ML Pipeline Generation via Functional Block Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Automated Machine Learning (AutoML) solutions are insufficient for non-expert data scientists to fully implement new machine learning projects due to scarcity of ML experts, as they offer only simplistic and partial solutions, and existing open-source software databases of ML projects are difficult to navigate and modify for new requirements.
Innovation Solution
A system that automatically searches and adapts existing ML projects by curating high-quality and diverse ML pipelines, identifying and indexing functional blocks, and synthesizing new pipelines using a search-and-adapt workflow, allowing non-expert data scientists to generate executable ML pipelines for new tasks and datasets without manual modification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If current AutoML solutions are used, then non-experts can perform basic ML tasks, but the solutions are simplistic and partial, insufficient for fully implementing new ML projects
Solution Approach 1:
The patent segments ML pipelines into functional blocks (data loading, preprocessing, modeling, evaluation) that can be independently selected and combined. This allows the system to handle diverse project requirements by assembling different block combinations while maintaining ease of use through automated pipeline generation.
Solution Approach 2:
The patent creates a universal pipeline generation system that can handle multiple ML project types and requirements through a single platform. The system uses a corpus of existing pipelines and functional blocks that can be adapted to various scenarios, providing both simplicity and versatility.
2Quantity of substance
If existing open-source software databases of ML projects are used, then a variety of ML projects are available, but they are difficult to navigate and modify for new requirements
Solution Approach 1:
The patent breaks down complex ML pipelines into discrete functional blocks with standardized interfaces. This segmentation makes the system easier to navigate and modify, as users can selectively combine blocks rather than navigating through monolithic codebases. The functional blocks are organized in a corpus that can be searched and filtered.
Solution Approach 2:
The patent introduces an intermediary system (pipeline generator) that sits between the user and the existing ML projects. This intermediary automatically searches the corpus, selects relevant functional blocks, and assembles pipelines, shielding users from the complexity of navigating and modifying existing code.
3Adaptability or versatility
If manual pipeline creation is used, then pipelines can be customized for specific requirements, but it requires significant expert knowledge and time investment
Solution Approach 1:
The patent performs preliminary action by pre-defining a corpus of functional blocks and pipelines that have been developed and tested in advance. Users can then search and combine these pre-prepared blocks rather than creating everything from scratch, significantly reducing creation time while maintaining customization capability through selective combination.
Solution Approach 2:
The patent merges existing functional blocks and pipelines to create new customized solutions. By combining pre-defined blocks in different configurations, the system achieves customization without requiring experts to build everything from scratch, reducing time investment while maintaining adaptability.
4Device complexity
If simplistic AutoML solutions are used, then the implementation process is simplified, but the solutions are partial and insufficient for complex projects
Solution Approach 1:
The patent segments the ML implementation process into distinct functional blocks (data loading, preprocessing, modeling, evaluation, deployment). This segmentation allows the system to provide complete, reliable implementations by ensuring each function is properly handled by appropriate blocks, while keeping the overall system manageable through modular architecture.
Solution Approach 2:
The patent creates a universal pipeline generation system that can handle both simple and complex project requirements through the same platform. The system's versatility allows it to provide complete, reliable implementations for complex projects while remaining simple enough for basic tasks, eliminating the trade-off between simplicity and completeness.
Data Source
AI summary
According to one or more embodiments, operations may include storing existing machine learning (ML) projects in a corpus. The operations may also include generating a search query for a new ML project based on a new dataset and a new ML task for the new ML project. In addition, the operations may include searching through the existing ML projects stored in the corpus, based on the search query, for a set of existing ML projects. Moreover, the operations may include merging the ML pipelines of the set of existing ML projects to generate a new ML pipeline for the new ML project. In addition, the operations may include adapting functional blocks of the new ML pipeline for the new ML project to enable the new ML pipeline to be executed to perform the new ML task on the new dataset.