The application discloses a large
language model weight
compression method, an
inference operation method, a
system and a medium, and belongs to the technical field of large
language model weight compression and
inference operation optimization. The method comprises the following steps: acquiring a weight matrix of a large
language model and constructing an exponential high-frequency window and a benchmark value, classifying and encoding each weight data into a fixed-length
code word and splitting the fixed-length
code word into an independent
bitmap, synchronously generating a compact value
stream and a
rollback value
stream, and packing the compact value
stream and the
rollback value stream into weight compression data; loading the weight compression data, generating a state
mask and a channel prefix
mask in a
register based on the
bitmap, calculating a reading offset by using a
population count instruction, reading a compressed
data segment from a corresponding
data stream, reconstructing weight data in the register in combination with the
code word, and finally directly inputting the weight data into a
matrix multiplication unit for operation; therefore, by implementing the application, the problem of efficiency reduction caused by
control flow divergence and redundant memory access in GPU
inference of existing variable-length encoding can be solved, and the
model inference efficiency under
lossless compression can be improved.