Neural Network Weight Discretizing Method for Portable Devices

The patent changes the parameter representation from continuous floating-point weights to discrete quantized weights with limited precision. By representing weights using a small number of bits (e.g., 1-bit, 2-bit, or 3-bit quantization), the computation complexity is significantly reduced while maintaining acceptable performance through optimized discrete weight updates and gradient approximation methods during training.

The patent segments the weight update process into discrete state transitions rather than continuous adjustments. By dividing the weight space into discrete levels and using stochastic gradient descent to transition between these discrete states, the method reduces computational overhead while preserving the essential learning capability of the neural network.

The patent introduces dynamic quantization where the number of discrete weight levels and precision can be adaptively adjusted during training based on the specific layer and importance of weights. This dynamic approach allows the system to use lower precision for less critical weights (reducing buffer overhead) while maintaining higher precision for important weights, thereby balancing computation complexity and memory requirements.

The patent applies partial quantization selectively to different parts of the neural network. Instead of uniformly quantizing all weights to binary values, the method applies different quantization levels to different layers or weight groups, using binary or low-precision quantization for convolutional layers where it benefits computation while maintaining higher precision in fully connected layers where accuracy is more critical, thus optimizing the trade-off between computation complexity and buffer overhead.

The patent fundamentally changes the precision parameter of weight representation from full floating-point precision to discrete quantized precision. By using limited-bit representations (1-bit, 2-bit, 3-bit) for weights during training and inference, the storage space consumption and power consumption are dramatically reduced while training accuracy is maintained through carefully designed quantization-aware training algorithms that compensate for the reduced precision.

The patent replaces the traditional floating-point arithmetic mechanism with discrete state transition mechanisms. Instead of using continuous floating-point weight updates that require high-precision storage and computation, the method uses discrete state transitions guided by gradient directions, substituting the floating-point mechanism with a discrete-state machine approach that is more energy-efficient and suitable for hardware implementation in portable devices.

The patent implements dynamic precision adjustment where the quantization level of weights can be adaptively changed during different training stages and for different network layers. Early training stages may use higher precision to establish accurate weight directions, while later stages and less critical layers use lower precision to maximize processing efficiency. This dynamic approach allows the system to optimize the trade-off between weight value precision and processing efficiency based on actual training needs.

The patent applies preliminary high-precision initialization and early training with higher quantization levels to establish accurate weight directions and network structure before transitioning to lower precision modes. This preliminary action ensures that the discrete quantized weights start from accurate positions, and the network structure is properly established before the benefits of reduced precision are fully exploited, thereby minimizing the loss of weight value precision while achieving improved processing efficiency.