The invention provides a key value cache scheduling method and
system for
language model inference, and is applied to the technical field of
video memory management and key value cache
multiplexing of a large-scale
language model inference engine. The method comprises the following steps: acquiring an input sequence, extracting a candidate
multiplexing fragment, calculating a fragment hash value of the candidate
multiplexing fragment, querying a corresponding historical
logic block hash list in a fragment index table, and generating a current
logic block hash list for the fragment in a pre-filling process, mapping the Hash of the current
logic block to the
physical address of the historical physical key value cache block through a prefix
Hash table to realize logic Hash
alias, and scheduling key value cache data in an underlying
video memory to participate in autoregressive decoding according to the logic Hash
alias; by means of the scheme, prefix tree matching limitation can be broken through, zero-copy cross-context multiplexing of semantic fragments at any position is achieved, pre-filling calculation redundancy and first word
delay are remarkably reduced, and GPU
video memory utilization efficiency and reasoning
throughput under a multi-agent concurrent scene are improved.