Accelerator kernel autotuner for sparse hyperparameter spaces

Abstract

Effective kernel hyperparameters can be determined by an autotuner configured explore the entire hyperparameter space. A main process of the autotuner divides the hyperparameter space into batches, while worker processes determine batch-leading hyperparameter combinations for the batches along with a performance metrics for each batch-leading hyperparameter combination. The batch-leading hyperparameter combination for a given batch can be determined using an optimization tool which receives performance metrics for an initial sample set of valid hyperparameter combinations of the given batch and applies an optimization technique to select additional hyperparameter combinations of the given batch for evaluation. The number of hyperparameter combinations in the initial sample set and the number of additional hyperparameters to select for evaluation can be input as tuning settings of the autotuner. The main process receives the batch-leading hyperparameter combinations for the batches and compares their performance metrics to determine an overall-leading hyperparameter combination for the kernel.

Images