Neural Network Stem Block Normalization for Machine Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network architectures for machine vision lack normalization layers, which hinders their accuracy and training efficiency, particularly in handling input image distributions and parameter initialization.
Innovation Solution
Incorporating multiple normalization layers in the stem block of a neural network architecture, including a patch layer, an embedding layer, and two normalization layers, to preprocess input images into normalized patches and vector embeddings before feeding them to a visual transformer block for machine vision tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional transformer models are used without normalization layers, then the device complexity is reduced, but the manufacturing precision (accuracy) deteriorates
Solution Approach 1:
The patent divides the neural network into distinct functional blocks (stem block with patch layer, embedding layer, and multiple normalization layers, followed by transformer blocks). This segmentation allows each component to perform its specific function optimally, with normalization layers specifically designed to stabilize training and improve accuracy without overwhelming the entire system complexity.
Solution Approach 2:
The patent applies normalization layers in advance (before the transformer blocks) to preprocess the image patches and embeddings. This preliminary normalization action stabilizes the input distribution for the transformer blocks, improving training accuracy and convergence before the main processing occurs.
2Productivity
If multiple normalization layers are added to the neural network, then the training efficiency improves, but the device complexity increases
Solution Approach 1:
The normalization layers are positioned at the beginning of the network (in the stem block) to perform preliminary normalization on raw image patches before they enter the main transformer processing. This preliminary action accelerates training convergence by ensuring stable input distributions from the start, rather than requiring extensive training to achieve stability.
Solution Approach 2:
The normalization layers act as intermediary components between the patch embedding layer and the transformer blocks. They mediate the transition by normalizing the embeddings, which stabilizes training and allows for higher learning rates, thereby improving training speed without requiring fundamental changes to the transformer architecture.
3Reliability
If normalization layers are incorporated into the stem block, then the reliability of training improves, but the device complexity increases
Solution Approach 1:
The patent segments the neural network architecture into distinct functional blocks, with normalization layers specifically placed within the stem block. This segmentation isolates the normalization function to where it is most needed (at the beginning of processing), improving training reliability without distributing complexity throughout the entire network.
Solution Approach 2:
The normalization layers serve as intermediary components that stabilize the training process by normalizing inputs at critical points in the stem block. They mediate between raw image patches and the transformer processing, ensuring reliable and stable training without requiring complex modifications to the core transformer architecture.
Data Source
AI summary
One example aspect of the present disclosure is directed to a neural network for machine vision. The neural network may include a stem block that includes a set of stem layers. The neural network may additionally include a visual transformer block. The set of stem layers may include a patch layer, a first normalization layer, an embedding layer, and a second normalization layer. The patch layer subdivides an input image into a set of image patches. The first normalization layer generates a set of normalized image patches by performing a first normalization process on each image patch of the set of image patches. The patch layer feeds forward to the first normalization layer. The embedding layer generates a set of vector embeddings. Each vector embedding of the set of embedding vectors is a projection of a corresponding normalized image patch from the set of normalized image patches onto a visual token. The first normalization layer feeds forward to the embedding layer. The second normalization layer generates a set of normalized vector embeddings by performing a second normalization process on each vector embedding of the set of vector embeddings. The embedding layer feeds forward to the second normalization layer. The transformer block enables one or more machine vision tasks for the input image based on the set of normalized vectors. The second normalization layer feeds forward to the transformer block.