Partially Shared Multi-Task DNNs for Scalable Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning techniques struggle to share model variables among multiple DNNs, especially when the cost function is non-convex, and conventional variable update rules face convergence issues during learning.
Innovation Solution
A neural network system with shared and unshared layers among DNNs, utilizing a constraint-based framework to enable variable sharing and optimize model variables across multiple DNNs, even when they have different tasks or are located on the same apparatus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple DNNs operate independently to handle different tasks, then each DNN can be optimized for its specific task, but the overall system complexity increases and computational resources are wasted
Solution Approach 1:
The patent merges multiple independent DNNs into a single multi-task DNN that shares common layers and model variables. The system integrates multiple task-specific processing paths under a unified architecture, where shared layers process common features for all tasks, reducing overall system complexity while maintaining task-specific performance through specialized output layers.
Solution Approach 2:
The patent creates a universal DNN architecture that performs multiple functions simultaneously. The shared model variables and common layers serve all tasks universally, while task-specific layers handle individual task requirements. This multi-functional design allows a single system to replace multiple independent DNNs, reducing complexity without sacrificing task-specific reliability.
2Reliability
If multiple DNNs are trained independently, then each DNN can achieve optimal performance for its task, but the training time and computational cost increase significantly
Solution Approach 1:
The patent combines multiple independent training processes into a single unified training procedure. The shared model variables are updated simultaneously for all tasks through a unified loss function that aggregates performance across all tasks. This merging of training processes reduces total training time compared to sequentially or independently training multiple separate DNNs, while maintaining model accuracy through the shared representation learning.
Solution Approach 2:
The patent enables continuous and simultaneous learning across all tasks through a unified training process. The shared layers continuously learn common patterns that benefit all tasks, while task-specific layers concurrently optimize for their individual objectives. This continuous multi-task learning approach is more time-efficient than discrete independent training while maintaining the accuracy benefits of task-specific optimization.
3Productivity
If model variables are shared among multiple DNNs, then computational efficiency improves and resources are utilized better, but convergence difficulties arise especially with non-convex cost functions
Solution Approach 1:
The patent implements dynamic variable sharing where the degree and pattern of sharing among tasks can adapt during training. The system dynamically adjusts how shared model variables are updated based on task-specific gradients and convergence status. This dynamic approach allows the system to maintain computational efficiency through sharing while avoiding convergence pitfalls by flexibly modulating the sharing intensity based on real-time training conditions.
Data Source
AI summary
Techniques for achieving a scalable DNN and a multi-task DNN, for example, are provided. A neural network system is a neural network system including a plurality of models. Each one of the plurality of models is a DNN including a plurality of layers. Some or all of the plurality of models include at least one layer for which some or all of model variables are equivalent or common (hereinafter referred to as a “shared layer”) and also include at least one layer for which the model variables are not equivalent nor common (hereinafter referred to as an “unshared layer”).