The invention relates to the technical field of reasoning optimization, in particular to a PD separation reasoning framework optimization method oriented to a large
language model, which comprises the following steps: S1, reconstructing a memory structure of a KV Cache, adjusting an original discrete
storage structure allocated according to a model layer into a continuous
storage structure allocated according to blocks, and changing the memory structure from [layer, k / v, block id, numhead, head,
block size] into [block id, layer, k / v, numhead,
head size,
block size]; s2, dividing a short sequence, a medium sequence and a long sequence according to the length of the cue word, combining the short sequence into a Batch group, and preferentially extruding the short sequence and then
processing the long sequence; and S3, deploying a
hybrid throughput node cluster, and according to the input prompt
word length dynamic allocation request, allocating a short request to a low
throughput TP node of the
throughput, and allocating a long request to a high throughput TP node of the throughput. The method is more suitable for a PD separated
system architecture, and the transmission efficiency of the KV cache and the calculation efficiency of the GPU are improved, so that the throughput of the
system is improved, and the maximum
resource utilization is achieved.