Low-Rank Locally Connected Neural Network Layers for Spatial Invariance Control
Solution Overview
Problem
Existing neural networks face limitations in object recognition accuracy due to their spatial invariance, which restricts their ability to recognize objects in varying spatial locations within an input.
Innovation Solution
The introduction of low-rank locally connected neural network layers that relax spatial invariance by convolving different kernels against specific portions of the input, allowing for a controlled degree of spatial invariance relaxation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If weight sharing between parameters is used to achieve spatial invariance, then spatial invariance is improved, but object recognition accuracy deteriorates
Solution Approach 1:
The patent applies local quality by allowing different portions of the input to have different degrees of spatial invariance. Specifically, it uses multiple kernels with different weights applied to different portions of the input, enabling the network to maintain spatial invariance where needed while breaking it where it improves recognition accuracy. This is achieved through the low-rank locally connected layer that selectively applies weight sharing based on the specific input portion being processed.
Solution Approach 2:
The patent implements dynamics by making the degree of spatial invariance adaptive rather than fixed. The system dynamically adjusts the level of weight sharing based on the input characteristics and task requirements. Through the low-rank decomposition and learned weight matrices, the network can transition between spatially invariant and spatially variant processing modes, optimizing performance for different objects and contexts.
2Measurement precision
If spatial invariance is relaxed to improve object recognition accuracy, then object recognition accuracy is improved, but the ability to recognize objects regardless of spatial location deteriorates
Solution Approach 1:
The patent changes the parameters of spatial invariance control by introducing a low-rank decomposition framework where the weight matrices can be adjusted continuously. By modifying the rank parameter and the specific weight values in the decomposition, the system can control the degree of spatial invariance relaxation. This allows fine-grained adjustment of the parameter to optimize both recognition accuracy and spatial invariance based on the specific task requirements.
Data Source
AI summary
A computing system can include one or more non-transitory computer-readable media that collectively store a neural network including one or more layers with relaxed spatial invariance. Each of the one or more layers can be configured to receive a respective layer input. Each of the one or more layers can be configured to convolve a plurality of different kernels against the respective layer input to generate a plurality of intermediate outputs, each of the plurality of intermediate outputs having a plurality of portions. Each of the one or more layers can be configured to apply, for each of the plurality of intermediate outputs, a respective plurality of weights respectively associated with the plurality of portions to generate a respective weighted output. Each of the one or more layers can be configured to generate a respective layer output based on the weighted outputs.