Data Tiering With Lifetime Data Valuation for Deep Learning I/O Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data valuation techniques assign fixed values to data points for a single training task, making them inapplicable for subsequent ML models, and existing training methods cause I/O bottlenecks and reduce model accuracy due to inefficient use of multi-tier storage systems and random data selection.
Innovation Solution
Implementing path-dependent data valuations that evolve across multiple training tasks and optimizing data tiering by using random and accuracy-based mini-epochs to reduce I/O bottlenecks and maintain model convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional data valuation techniques are used to assign fixed values to data points for a single training task, then data valuation can be performed for that specific task, but the valuation becomes inapplicable for subsequent ML models and training tasks
Solution Approach 1:
The patent transforms static fixed data valuations into dynamic path-dependent valuations that evolve across multiple training tasks. The valuation of each data point is updated iteratively based on its contribution to model performance in each task, allowing the same data point to have different valuations depending on the training path and task context.
Solution Approach 2:
The system implements feedback loops where model performance metrics from each training task are used to update data valuations for future tasks. The valuation update mechanism uses performance feedback to adjust which data points are considered high-value for subsequent training, creating an adaptive valuation system that learns from past training outcomes.
2Productivity
If random data selection and traditional training methods are used, then training can proceed with simple data access, but I/O bottlenecks occur and model accuracy is reduced due to inefficient use of multi-tier storage systems
Solution Approach 1:
The system performs preliminary actions by pre-loading high-value data points identified through path-dependent valuation into faster storage tiers before training begins. This proactive data preparation ensures that critical data is readily available during training, preventing I/O bottlenecks and maintaining training throughput while improving model accuracy.
Solution Approach 2:
The patent applies local quality by differentiating data access strategies based on data value. High-value data points identified through valuation are stored in faster storage tiers with priority access, while lower-value data remains in slower tiers. This localized optimization of storage quality for critical data points resolves the contradiction between throughput and accuracy.
3Ease of manufacture
If all data samples are treated equally in training datasets, then data processing is simplified, but model performance is suboptimal when the dataset contains incorrect labels or varying sample quality
Solution Approach 1:
The system changes the parameter of data weighting from uniform to variable based on path-dependent valuation. Each data point is assigned a weight that reflects its proven value across training tasks, allowing the training process to automatically prioritize high-quality samples and downweight or exclude problematic samples without manual intervention.
Data Source
AI summary
Systems and methods are configured to provide lifetime data valuations for a dataset that evolves across multiple machine learning training tasks by providing and updating path-dependent data valuations for data points in the dataset during each training task. A current machine learning training task may include splitting the dataset into multiple random mini-epochs and training the current machine learning model using a first random mini-epoch and an accuracy mini-epoch, which consists of high value data points from the path-dependent data valuations. The random and accuracy mini-epochs can be, during the training, iterated for a number of times during the training, while a second random mini-epoch is prefetch. During the training, the path-dependent data valuations can be updated based on data valuations during the current training and a similarity between the current machine learning model and prior trained machine learning models.