The application discloses a large
language model inference parameter
automatic tuning method and
system, comprising the following steps: a, receiving a user end tuning task to construct a task
queue; b, calculating the required total number of GPUs for each experiment of a
processing task, screening a target GPU set for each experiment according to GPU storage
resource information, if successful, executing step d, otherwise, executing step c; c, judging whether the
waiting time of the experiment is less than a task
waiting time threshold, if yes, returning to step b after a preset time, otherwise, marking the current experiment as FAILED and executing step e; d, testing the experiment by using an LLM
inference performance test tool and calculating a service level
index score of the experiment according to a test result; e, releasing the GPU, judging whether the maximum number of experiments is reached, if yes, selecting an optimal experiment according to the service level index scores of all experiments of the current task, otherwise, returning to step b to execute the next experiment of the task. The application can effectively reduce the tuning time.