Embodiments of the present application provide an end-side
model inference dynamic optimization method, device, equipment and medium. The method surrounds the end-side single-flow CPU
inference path, establishes a performance,
energy consumption and
thermal state measurement
system covering the prompt
processing stage and the self-recurrence generation stage, and adopts a fixed word window unified index statistics and configuration decision. The offline measurement is performed on the CPU frequency, thread number and affinity
configuration space, the feasible boundary and preferred priori are refined, and a discrete
neighborhood search dynamic optimization method under the offline priori constraint is proposed. Thus, taking a unified comprehensive cost function as an optimization target, the
online search is limited in the feasible neighborhood obtained through offline screening under the first word
delay,
thermal safety and frequency
accessibility constraint, and combined with the minimum
residence, anti-shake, limited memory and controlled jump-out mechanism, the low-overhead joint adjustment of the CPU frequency and thread number is realized, while the adaptability to state changes is improved.