Large Model Predictor for Deep Learning Training Optimization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Determining an optimal training approach for large deep learning models is challenging due to the exponential number of choices and considerations affecting training parameters, making traditional methods ineffective and resource-intensive.

Innovation Solution

A computer-implemented method that identifies model and system characteristics to automatically determine an optimal training approach using a trained large model predictor, adjusting system parameters and settings for improved efficiency and reduced training time.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If traditional trial and error methods are used to determine training approach, then comprehensive exploration of training parameters is possible, but resource consumption increases and training time extends

Engineering Contradiction:

Improvetraining parameter optimizationVSAvoidtraining time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary analysis of model characteristics and system configurations before actual training begins. A trained large model predictor is used to predict optimal training approaches in advance, avoiding the need for trial-and-error experimentation during the training process itself. This preliminary determination of training parameters significantly reduces the time required during actual model training.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A trained large model predictor serves as an intermediary between the model characteristics/system configurations and the training approach determination. This intermediary component processes the input characteristics and outputs recommended training approaches, eliminating the need for direct trial-and-error testing and reducing resource consumption while maintaining optimization quality.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If manual determination of training approach is performed, then flexibility in adjusting parameters is maintained, but effectiveness decreases due to inability to consider all characteristics

Engineering Contradiction:

Improveparameter adjustment flexibilityVSAvoidtraining approach effectiveness

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system automatically determines optimal training approaches by itself, using the trained large model predictor to analyze model characteristics and system configurations. This self-service capability eliminates the need for manual parameter adjustment while considering all relevant characteristics comprehensively, thereby improving reliability without sacrificing adaptability.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system changes from manual parameter specification to automated parameter determination based on model characteristics and system configurations. The trained large model predictor dynamically adjusts training parameters based on the specific characteristics of each model and system, providing both effectiveness through comprehensive consideration and adaptability through characteristic-based customization.

Inventive Principle:

Principle #35Parameter changes

3Manufacturing precision

If exponential number of training choices are explored, then optimal training approach can be found, but resource intensity increases significantly

Engineering Contradiction:

Improvetraining approach optimizationVSAvoidcomputational resource consumption

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The system extracts only the essential model characteristics and system configurations that are most relevant to training approach determination. Instead of exploring all possible training choices, the trained large model predictor identifies and processes only the critical features, significantly reducing computational resource consumption while maintaining optimization quality.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The trained large model predictor performs preliminary filtering and analysis of training choices before actual training begins. By predicting the optimal training approach in advance based on characteristic analysis, the system avoids the need to explore the exponential number of possible training configurations, thereby reducing resource intensity while achieving optimal results.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20210158147A1Training approach determination for large deep learning models

Publication Date: 2021.05.27 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

In an approach to determining an optimal training approach for a large deep learning model based on model characteristics and system characteristics. The one or more computer processors identify one or more model characteristics associated with a deep learning model. The one or more computer processors identify one or more system configurations associated with a system training the deep learning model. The one or more computer processors determine a training approach for the deep learning model utilizing a trained large model predictor fed with the one or more identified model characteristics and the one or more identified system configurations. The one or more computer processors train the deep learning model utilizing the determined training approach.