Abstract Data Types for Reproducible Machine Learning Workflows
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning workflow management systems lack reproducibility, data standardization, and efficient debugging capabilities, restricting agility and scalability in complex machine learning workflows.
Innovation Solution
A method and system that transform abstract data sets into abstract data types, generate abstract pipelines, and implement them as packages, allowing for caching, reproducibility, and continuous integration, with an orchestrator managing task execution and providing meta constructs for hyperparameter tuning and debugging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional workflow managers are used to manage machine learning workflows, then multi-language support and declarative workflows are provided, but reproducibility, data standardization, and efficient debugging capabilities are lacking
Solution Approach 1:
The patent introduces an intermediary layer (abstract data types and execution plans) between the workflow definition and actual execution. This intermediary layer standardizes data representations and execution sequences, ensuring that workflows can be reproduced reliably across different languages and platforms while maintaining the adaptability benefits of conventional workflow managers.
Solution Approach 2:
The patent transforms workflow parameters from opaque, language-specific implementations to standardized, visible parameters with explicit data types and execution plans. This parameter transformation enables reproducibility by making all execution parameters explicit and standardized, while still allowing the system to accept multiple programming languages through the standardized parameter interface.
2Ease of operation
If complex machine learning workflows are implemented with fine-grained debugging requirements, then debugging capability is improved, but system complexity increases
Solution Approach 1:
The patent segments machine learning workflows into discrete, independently executable tasks with explicit input-output data types. Each task can be debugged individually with fine-grained control, while the overall workflow complexity is managed through the standardized task interface. This segmentation enables efficient debugging without proportionally increasing overall system complexity.
Solution Approach 2:
The execution plan acts as an intermediary that provides fine-grained debugging capabilities without exposing the full complexity of the underlying system. It translates high-level workflow definitions into detailed execution sequences that can be inspected and debugged step-by-step, while hiding the complexity of task orchestration and resource management.
3Stability of the object's composition
If machine learning workflows require standardization and re-usability, then workflow management is improved, but agility in business requirement changes is reduced
Solution Approach 1:
The patent creates a dynamic system where standardized abstract data types and task definitions can be flexibly reconfigured through execution plans. The standardized components maintain stability and re-usability, while the execution plan layer enables dynamic adaptation to changing business requirements by reordering, filtering, or modifying task execution sequences without changing the underlying standardized definitions.
4Productivity
If caching of executed task outputs is implemented, then experiment agility is improved, but memory resources are consumed
Solution Approach 1:
The patent implements caching with local quality by caching task outputs based on their specificity and re-use potential. Instead of uniformly caching all outputs, the system selectively caches results based on their data types, task parameters, and expected re-usability in similar experiments. This approach improves experiment agility for frequently re-run tasks while minimizing memory consumption for unique or rarely re-used results.
Data Source
AI summary
A method and system for managing reproducible machine learning workflows are disclosed. The method includes receiving input comprising abstract data sets, and transforming abstract data sets into abstract data types. The method includes generating abstract pipelines using abstract data types, and implementing abstract pipelines as packages. The method includes configuring packages as map of key-value pairs comprising keys, and storing configured packages in database. The method includes generating execution plan by converting abstract pipelines from the configured packages into concrete pipelines. Further, method includes transmitting execution plan to orchestrator to merge individual concrete pipelines into dataset dependency graph, and to mark tasks in dataset dependency graph. The method includes executing tasks as cluster, by calling appropriate command, and obtaining predictions from different models or same model with different hyperparameters to provide meta construct, upon executing tasks as cluster. The method includes outputting modified DAG comprising tasks mapped to configuration.