2 results about "Stochastic matrix" patented technology

This invention discloses a residual routing constraint method and system for enhancing the training stability of neural networks, comprising the following steps: S1, based on the residual routing constraint in the neural network... l Input feature tensor of the layer x l S1. Generate the original values ​​of the pre-scaling matrix, the post-scaling matrix, and the residual routing matrix; S2. Apply non-negativity constraints to and to obtain the pre-scaling matrix. H pre and post-scaling matrix H post By constraining optimization to satisfy the double stochastic matrix condition, the residual routing matrix is ​​obtained. H res S3, Utilize H pre right x l Premixing is performed to obtain preprocessed characteristics. u l ;Will u l Enter to the number l Layer core calculation function F layer Calculations are performed to obtain the layer output features. y l ;Will H post Acting on y l The scaled output will be obtained. H res Acting on x l After obtaining the stable residual, the scaled output is summed with the stable residual to obtain the first... l Layer output features x l +1. This invention significantly improves the numerical stability of deep neural network training.

Disclosed are a stochastic matrix-vector multiply-accumulate operation system and an operation method thereof. The operation system includes an input bit-stream generator, a weight bit-stream generator and a memory array. The input bit-stream generator generates a pulse sequence representing a corresponding input stochastic bit-stream according to respective preset input values of a preset input vector, the weight bit-stream generator generates a pulse sequence representing a corresponding weight stochastic bit-stream according to respective preset weight values of a preset weight matrix, and the pulse sequence representing the input stochastic bit-stream and the pulse sequence representing the weight stochastic bit-stream are applied to a word line and a bit line of the memory array respectively. In addition, a result multiply-accumulating between the input stochastic bit-stream and the weight stochastic bit-stream of each memory device of the memory array is stored as a conductance value of the corresponding memory device.