Model Zoo Ensemble for Multi-Task Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-task and continual learning methods fail to effectively leverage data from multiple tasks due to competition between tasks, leading to suboptimal performance and forgetting of previously learned information.
Innovation Solution
The Model Zoo algorithm iteratively grows an ensemble of small models, each trained on a subset of tasks, allowing for automatic selection of tasks based on their empirical risk, and maintains models from previous rounds to avoid competition and enhance data utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multiple tasks are trained together using existing multi-task learning methods, then data utilization is improved, but task competition causes performance degradation and catastrophic forgetting
Solution Approach 1:
The patent divides the learning process into multiple boosting rounds, where each round trains on a subset of tasks rather than all tasks simultaneously. This segmentation reduces task competition while still leveraging data from multiple tasks across different rounds, resolving the contradiction between data utilization and performance stability.
Solution Approach 2:
The patent maintains and preserves models from previous boosting rounds, effectively performing preliminary action by storing learned knowledge before new training occurs. This prevents catastrophic forgetting while allowing new tasks to be learned, addressing both data utilization and performance stability.
2Device complexity
If a single model is used for multiple tasks, then model complexity is reduced, but task competition leads to suboptimal performance on individual tasks
Solution Approach 1:
The patent creates multiple models through iterative boosting rounds, with each model specialized for specific task subsets. This segmentation allows each model to achieve high per-task accuracy without the complexity of a single large multi-task model, as each model remains relatively simple but collectively they cover all tasks.
Solution Approach 2:
The patent combines multiple simple models into an ensemble that collectively handles all tasks. This merging approach achieves the benefits of task-specific optimization while keeping individual model complexity low, resolving the contradiction between model complexity and per-task accuracy.
3Adaptability or versatility
If all tasks are trained incrementally in a single continual learning process, then adaptability is improved, but catastrophic forgetting of previously learned tasks occurs
Solution Approach 1:
The patent performs preliminary action by preserving and maintaining models from previous boosting rounds before training on new tasks. This ensures that previously learned information is not lost, while still allowing the system to adapt to new tasks through iterative model creation.
Solution Approach 2:
The patent discards the limitation of single-model continual learning by creating multiple models across boosting rounds, effectively recovering from catastrophic forgetting through the ensemble of preserved models while maintaining adaptability to new tasks.
4Manufacturing precision
If task-specific models are trained in isolation, then per-task accuracy is maximized, but data from multiple tasks is not leveraged
Solution Approach 1:
The patent merges the benefits of task-specific training with multi-task data utilization by creating an ensemble of models through boosting rounds. Each model is trained on specific task subsets (maintaining per-task accuracy) while collectively leveraging data from all tasks across rounds (improving data utilization).
Data Source
AI summary
Methods, systems, and computer readable media for machine learning of multiple tasks. In some examples, a method includes performing multiple rounds of training. For each training round, the method includes selecting a subset of computing tasks from the tasks being learned; building a feature generator for the subset of computing tasks; and training a task-specific classifier for each computing task, resulting in model for each computing task of the subset of computing tasks. The method can then include using the models for performing one of the computing tasks.