Hyperparameter Tuning Resource Allocator Using Tensor Swapping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep learning hyperparameter tuning requires large-scale accelerator clusters, which are often inaccessible or cost-prohibitive for data scientists and researchers, leading to resource crunches and inefficiencies in hyperparameter search and tuning processes.

Innovation Solution

A hyperparameter tuning resource allocator manages limited computing resources to perform parallel experiments using tensor swapping, allowing dynamic adjustment of accelerator usage and simulating large-scale workloads, thereby enabling hyperparameter search and tuning on cloud environments with usual accelerator resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If large-scale accelerator clusters are used for hyperparameter tuning, then hyperparameter search capability is improved, but infrastructure cost and resource requirements increase

Engineering Contradiction:

Improvehyperparameter search capabilityVSAvoidaccelerator resources

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The system segments the hyperparameter search process into multiple parallel experiments that can be distributed across available accelerators. Each accelerator works on a portion of the search space, enabling scalable hyperparameter tuning without requiring a single large cluster. The resource allocator divides the workload into manageable chunks that can be executed independently on individual accelerators or smaller groups.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The hyperparameter tuning resource allocator serves multiple functions: it manages accelerator allocation, performs workload batching, coordinates parallel experiments, and optimizes resource utilization. This multi-functional system replaces the need for dedicated large-scale clusters by making available accelerators versatile and adaptable to hyperparameter search workloads.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Speed

If more accelerators are allocated for hyperparameter tuning, then tuning speed increases, but resource availability and accessibility decrease

Engineering Contradiction:

Improvetuning speedVSAvoidresource accessibility

Core Design Contradiction:

SpeedVSAdaptability or versatility

Solution Approach 1:

The resource allocation system is dynamic and adaptive, adjusting accelerator allocation based on available resources and workload requirements. The hyperparameter tuning resource allocator monitors resource availability and dynamically scales the number of parallel experiments and batch sizes, enabling fast tuning when resources are abundant and graceful degradation when resources are limited.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes key parameters such as batch size and number of parallel experiments based on available accelerator resources. When more accelerators are available, the system increases parallel experiments and adjusts batch sizes to maximize tuning speed. When resources are constrained, it reduces these parameters while maintaining accessibility for users with limited resources.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If parallel experiments are increased for hyperparameter search, then search throughput improves, but workload management complexity increases

Engineering Contradiction:

Improvesearch throughputVSAvoidworkload management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The hyperparameter tuning resource allocator acts as an intermediary between the hyperparameter search algorithm and the accelerator cluster. It manages the complexity of coordinating parallel experiments by handling workload distribution, resource allocation, and experiment coordination centrally, shielding users from the underlying complexity while enabling high throughput through parallel execution.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Speed

If batch size is increased for each experiment, then convergence speed improves, but memory requirements and resource consumption increase

Engineering Contradiction:

Improveconvergence speedVSAvoidresource consumption

Core Design Contradiction:

SpeedVSUse of energy by moving object

Solution Approach 1:

The system uses partial action by implementing workload batching where each accelerator processes a portion of the total batch size. Instead of requiring each individual experiment to use large batch sizes for fast convergence, the system achieves equivalent throughput by distributing smaller batches across multiple parallel experiments, reducing memory requirements while maintaining convergence speed.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11409564B2Resource allocation for tuning hyperparameters of large-scale deep learning workloads

Publication Date: 2022.08.09 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A method, computer system, and a computer program product may perform hyperparameter tuning. Hyperparameter search input data may be received. A search algorithm to generate a set of hyperparameters may be invoked. A hyperparameter tuning resource allocator may receive physical resource data regarding a cluster. The hyperparameter tuning resource allocator may determine a required workload and may use a received number of parallel experiments to determine the required workload. The hyperparameter tuning resource allocator may determine whether an option to mandate the workload batch size was selected. The cluster may perform the number of parallel experiments to search and tune the set of hyperparameters. The hyperparameter tuning resource allocator may manage the cluster according to the physical resource data and the required workload and by using tensor swapping.