This invention discloses a dynamic memory compression and layer scheduling method,
system, and portable AI device based on sparse awareness. The method includes: analyzing the
impact of each weight of a trained
large model on accuracy, dividing it into key weights and secondary weights and performing asymmetric quantization compression; predicting the access probability of each model layer, performing a prefetch operation in parallel during the current model layer's computation to obtain the predicted subsequent model layer weights after asymmetric quantization compression; storing them in off-
chip main memory and completing
bus transmission in a compressed state, during which only valid weights are loaded using a pre-generated sparse
mask; decompressing the data at the computation
entry point; and using a sliding window strategy to dynamically manage the swapping in and out of the model
layers: extracting the weight data of the subsequent model
layers from the decompressed weight data and moving it into a window in the on-
chip main memory, and then sending it to the physical execution engine of the
large model to perform
inference, thereby achieving stable operation of a large parameter model on a small-capacity memory device.