This invention discloses a method for accelerating
inference of multimodal large models based on KV Cache low-bit quantization, relating to the fields of
artificial intelligence and
deep learning. The method includes: a pre-filling stage, in which a subspace is constructed by querying the
singular value decomposition of the
tensor; iterative quantization with orthogonal constraints applied to the key
tensor, and token-by-token quantization of the value
tensor, both compressed to 2 bits; treating the weights as complex numbers, and quantizing the phase to the fourth root of unity;
inference, layer normalization of the visual
encoder is fused; in the decoding stage, token-by-token generation is performed, key-value quantization is carried out in real time and stored in the cache, and attention calculation is completed by combining the history. This invention can significantly reduce memory usage, improve
inference throughput, fully utilize the 2-bit storage space, maintain model accuracy, and eliminate the need for additional fine-tuning or calibration data.