Replicated Neural Network Layers for Sparse Weight Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly deep neural networks, face challenges in maintaining accuracy when using sparse weight values, especially in layers with fewer weight values than typical, leading to potential losses in network performance.
Innovation Solution
The approach involves duplicating or triplicating certain layers of a machine-trained network and assigning different sets of possible weight values to the replica layers, allowing for increased complexity and accuracy while maintaining sparsity and compatibility with specialized inference circuits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sparse weight values are used in neural network layers, then device complexity is reduced and inference speed is improved, but manufacturing precision (accuracy) deteriorates
Solution Approach 1:
The patent divides a single neural network layer into multiple replica layers (e.g., three replica layers), each processing the same input data independently with sparse weight values. The outputs of these replica layers are then combined through element-wise addition to produce the final output. This segmentation allows each replica layer to maintain sparsity for fast inference while the combination of multiple replicas preserves or enhances accuracy by providing redundant computational paths.
2Manufacturing precision
If the number of possible weight values is increased, then manufacturing precision (accuracy) is improved, but device complexity increases
Solution Approach 1:
The patent creates multiple copies (replica layers) of the same neural network layer structure with sparse weight values drawn from a limited discrete set (e.g., {0, 1, -1}). Instead of using a single layer with complex continuous weight values, the system uses multiple simpler replica layers whose outputs are combined. This copying approach achieves effective higher precision through the aggregation of multiple discrete-valued computations while keeping each individual replica layer simple and compatible with specialized inference hardware.
Data Source
AI summary
Some embodiments provide a method for executing a layer of a neural network, for a circuit that restricts a number of weight values used per layer. The method applies a first set of weights to a set of inputs to generate a first set of results. The first set of weights are restricted to a first set of allowed values. For each of one or more additional sets of weights, the method applies the respective additional set of weights to the same set of inputs to generate a respective additional set of results. The respective additional set of weights is restricted to a respective additional set of allowed values that is related to the first set of allowed values and the other additional sets of allowed values. The method generates outputs for the particular layer by combining the first set of results with each respective additional set of results.