AI Model Lightweighting via Pruning and Quantization

The patent segments the AI model into multiple layers and applies different processing strategies to each layer. Specifically, it divides the model into convolutional layers, pooling layers, and fully connected layers, applying pruning and quantization selectively to different layers based on their sensitivity to these operations. This allows the model to maintain accuracy in critical layers while achieving speedup in less sensitive layers.

The patent changes the parameter precision of the AI model by applying quantization, which converts high-precision floating-point parameters (32-bit) to low-precision integer parameters (8-bit or 4-bit). This parameter change reduces computational complexity and memory bandwidth requirements, thereby improving inference speed while maintaining detection accuracy through careful selection of quantization strategies and calibration processes.

The patent replaces expensive, complex hardware (GPUs with high precision floating-point units) with cheaper, simpler hardware that can execute quantized models. By converting the model to use low-precision integers, the system can run on microcontrollers or simple processors that are orders of magnitude cheaper than GPUs, while maintaining acceptable detection accuracy through the pruning and quantization process.

The patent substitutes the mechanical/computational system of high-precision floating-point arithmetic with a simpler integer arithmetic system. This substitution replaces complex hardware requirements (GPUs, FPGAs) with simpler processors that can perform integer operations, significantly reducing device complexity and cost while maintaining model functionality through the quantization process.

The patent applies preliminary training to the pruned model after removing neurons or connections. This retraining process allows the model to recover accuracy lost during pruning by adjusting the remaining parameters, ensuring that the final pruned and retrained model achieves both speed improvement and maintained accuracy. The preliminary action of training before final deployment compensates for the accuracy loss from structural removal.

The patent uses accuracy feedback to guide the pruning process. By monitoring detection accuracy during and after pruning, the system can iteratively adjust which neurons or connections to remove, ensuring that pruning does not exceed the tolerance threshold for accuracy degradation. This feedback mechanism allows the optimization of inference speed while maintaining acceptable accuracy levels.