Machine Learning Model Lineage Graph for Storage Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems fail to effectively manage and track provenance information across machine learning models, leading to difficulties in debugging, updating, and storage optimization, particularly in ecosystems where ML models are interrelated and share structure and parameter values.
Innovation Solution
The implementation of a lineage graph abstraction that tracks provenance information across machine learning models using nodes and edges, allowing for automated updating and storage optimization through content-based hashing and delta compression, reducing storage footprint and enabling efficient model management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If provenance information is tracked across interrelated machine learning models, then model management and debugging efficiency is improved, but system complexity increases due to the need to maintain lineage graphs and provenance metadata
Solution Approach 1:
The patent introduces a lineage graph as an intermediary data structure that mediates between machine learning models and provenance tracking requirements. The lineage graph stores nodes representing models and edges representing provenance relationships, acting as a mediator that enables efficient tracking without directly complicating the model management system itself
Solution Approach 2:
The patent creates a copied representation of model relationships through the lineage graph, which is a simplified abstraction that mirrors the complex interrelationships between ML models. This copied structure enables efficient querying and tracking without requiring direct manipulation of the actual model dependencies
2Reliability
If all machine learning model checkpoints are stored independently, then model accuracy and completeness are preserved, but storage requirements increase significantly
Solution Approach 1:
The patent merges the storage of provenance information with the model checkpoint storage by embedding provenance metadata directly within the checkpoint data structure. This consolidation allows the system to track model relationships and ancestry information without requiring separate storage infrastructure
Solution Approach 2:
The lineage graph nodes serve multiple functions simultaneously: they store model checkpoint data, maintain provenance metadata, enable debugging traceability, and support model reconstruction. This multi-functionality reduces the need for separate systems for each purpose
3Device complexity
If provenance information is not tracked across machine learning models, then storage and processing overhead is reduced, but debugging and updating capabilities deteriorate
Solution Approach 1:
The patent performs preliminary action by automatically capturing and storing provenance information at the time of model creation and transformation, rather than requiring retroactive tracking. This preliminary capture of metadata enables efficient debugging without adding overhead during model execution or analysis phases
Data Source
AI summary
The present disclosure relates to methods and systems that create a lineage graph that tracks provenance information across machine learning models. The methods and systems use the lineage graph to facilitate machine learning model testing, diagnostics, and updating. The methods and system also use the lineage graph to determine a storage optimization for reducing a storage footprint of the machine learning models.