Multi-task Neural Network Training via Conditional Batch Mining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-task machine learning systems often become unbalanced during training, where one task may dominate the others, leading to suboptimal performance across all tasks.
Innovation Solution
The method involves using batch miners to select samples from different datasets for each task, assessing the cost of processing these samples, and adding them to batches if the cost falls within specified thresholds. Once a batch is complete, it is used to update the neural network, ensuring that each task is trained at a balanced level.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If iterative ML training technique is used with multiple tasks, then the ML system can learn multiple tasks using shared parameters, but one task may dominate the others leading to unbalanced performance
Solution Approach 1:
The patent segments the training process into separate batches for each task, with batch miners independently selecting samples for each task. This segmentation prevents task dominance by ensuring each task receives dedicated training attention in isolated batches, while the shared backbone maintains multi-task capability.
Solution Approach 2:
The patent implements feedback mechanisms where loss values from each task are monitored and used to adjust batch mining strategies. When a task shows signs of dominance or underperformance, the feedback loop modifies sample selection criteria to balance overall task performance across the multi-task system.
2Reliability
If manual hand tuning techniques are used to balance tasks, then task performance can be optimized, but the complexity increases significantly as the number of tasks increases
Solution Approach 1:
The patent employs self-service mechanisms where batch miners automatically adjust sample selection based on task-specific loss values and predefined thresholds. The system self-regulates balance without manual intervention by monitoring task performance and adapting batch composition dynamically, eliminating the need for complex manual tuning.
Solution Approach 2:
The patent changes training parameters dynamically through automated batch mining strategies. Instead of manual tuning, the system adjusts sample selection parameters based on real-time loss monitoring, transforming the training process from static manual control to dynamic automated parameter adjustment.
3Reliability
If samples are selected based on cost thresholds, then tasks are challenged at comparable levels, but the batch mining process adds complexity to sample selection
Solution Approach 1:
The patent performs preliminary actions by pre-defining cost thresholds and loss value ranges for each task before training begins. Batch miners use these pre-established criteria to automatically filter and select appropriate samples, simplifying the selection process while ensuring balanced task challenge levels.
Solution Approach 2:
The patent introduces batch miners as intermediary components between the datasets and the neural network. These intermediaries handle the complex sample selection logic by filtering samples through cost threshold criteria, shielding the main training system from the complexity of sample evaluation while ensuring balanced task challenges.
Data Source
AI summary
Methods, systems, and computer program products, for training a multitask neural network. First and second datasets are provided, containing samples for a first task and a second task, respectively. First and second batch miners are provided for mining samples from the first and second datasets. First and second costs for completing the first and second tasks, respectively, are assessed using a first sample mined by the first batch miner from the first dataset and a second sample mined by the second batch miner from the second dataset. When the first or second cost, respectively, falls within a range delimited by lower and upper thresholds, the is added to a first or second batch, respectively. When a termination condition is reached for either the first or second batch, the first or the second batch is used to update the neural network.