Neural Network Parameter Projection for Sparse Weight Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly deep neural networks (DNNs), face challenges in reducing the number of weights, especially in larger layers, which can lead to inefficiencies in computation and resource usage.
Innovation Solution
Decompose initial layers of a neural network into two successive layers, applying constraints to set scale values to zero, and insert activation functions, while using techniques like ADMM and probabilistic projection to enforce structural sparsity and weight quantization, reducing the number of weight values and improving computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of weights in deep neural network layers is reduced, then computational efficiency and resource usage improve, but the network's predictive accuracy and modeling capability deteriorate
Solution Approach 1:
The patent segments the weight matrix into multiple sub-matrices or blocks, allowing selective pruning of less important blocks while preserving critical ones. This segmentation enables fine-grained control over weight reduction, maintaining accuracy in essential regions while achieving compression in others.
Solution Approach 2:
The patent applies different pruning strategies and retention thresholds to different regions or blocks of the weight matrix based on their importance. Critical regions maintain higher weight density while less critical regions undergo more aggressive pruning, achieving local optimization of the accuracy-compression tradeoff.
2Adaptability or versatility
If deep neural networks use more weights in larger layers, then modeling capability improves, but computational complexity and resource consumption increase
Solution Approach 1:
The patent divides large layers into multiple smaller sub-layers or blocks, each with its own weight matrix. This segmentation reduces the computational burden of any single operation while maintaining the overall modeling capability through the collective contribution of all blocks.
Solution Approach 2:
The patent transforms the weight reduction problem from reducing the number of weights within a single layer to adding a new dimension of block structure. By organizing weights into multiple blocks across an additional dimension, the network achieves equivalent or better modeling capability with reduced per-layer complexity.
3Quantity of substance
If structural sparsity techniques are applied to remove weight values, then the number of weights decreases, but training complexity and convergence difficulty increase
Solution Approach 1:
The patent performs preliminary identification and marking of weights for pruning before actual removal. This preliminary action involves computing importance metrics for all weights, selecting which ones to prune, and preparing the network structure in advance, which simplifies the subsequent training process compared to iterative pruning approaches.
Solution Approach 2:
The patent introduces intermediary parameters or auxiliary networks that mediate the pruning process. These intermediaries help guide weight removal while maintaining training stability, acting as a buffer between the sparsity constraint and the main training objective to reduce convergence difficulty.
Data Source
AI summary
Some embodiments provide a method for training a machine-trained (MT) network. The method receives a network comprising a plurality of parameters. The method trains the network by iteratively (i) propagating inputs through the network to generate outputs and adjusting the parameters based on differences between the generated outputs and expected outputs to minimize a loss function with respect to the parameters, (ii) probabilistically projecting the parameters to minimize the loss function with respect to a set of constraints on the weight values, the probabilistic projection treating the parameters as probability distributions, and updating a set of variables of the loss function based on the probability distributions.