Neural Network Design Optimization via Precomputed Gradient Functions

Solution Overview

Problem

Current automated neural architecture search (NAS) techniques are inefficient and computationally expensive when optimizing neural network designs over multiple design parameters, such as layer width, weight bitwidth, and network depth, as they require validation of numerous architecture candidates.

Innovation Solution

A process is introduced to select design options for neural network features by computing function values based on sample weights and coefficients, using gradient functions and loss functions to optimize design decisions across multiple layers, with constraints on memory and latency, employing Monte Carlo methods for efficient optimization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If current automated neural architecture search techniques are used to optimize neural network designs over multiple design parameters, then design optimization can be achieved, but computational resources and time are excessively consumed

Engineering Contradiction:

Improvedesign optimizationVSAvoidcomputational time

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-computing gradient functions and loss function values for multiple design parameters before actual architecture search. This allows the system to have optimization guidance ready in advance, significantly reducing the computational time required during the actual neural architecture search process while maintaining design optimization quality

Inventive Principle:

Principle #10Preliminary action

2Manufacturing precision

If current automated neural architecture search techniques are used to optimize neural network designs over multiple design parameters, then design optimization can be achieved, but computational resources are excessively consumed

Engineering Contradiction:

Improvedesign optimizationVSAvoidcomputational resources

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The patent applies preliminary action by pre-computing gradient functions and loss function values for multiple design parameters before actual architecture search. This allows the system to have optimization guidance ready in advance, significantly reducing the computational time required during the actual neural architecture search process while maintaining design optimization quality

Inventive Principle:

Principle #10Preliminary action

3Manufacturing precision

If numerous architecture candidates are validated to optimize neural network design, then design quality can be improved, but computational complexity increases

Engineering Contradiction:

Improvedesign qualityVSAvoidcomputational complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent introduces gradient functions and loss functions as intermediary components that guide the architecture search process. These intermediaries provide directional guidance for optimization, allowing the system to evaluate and select architecture candidates more efficiently without requiring exhaustive validation of numerous candidates, thus reducing computational complexity while maintaining design quality

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20230042271A1System, devices and/or processes for designing neural network processing devices

Publication Date: 2023.02.09 ARM LTD

AI summary

Example methods, apparatuses, and/or articles of manufacture are disclosed that may be implemented, in whole or in part, using one or more computing devices to select options for decisions in connection with design features of a computing device. In a particular implementation, design options for two or more design decisions of neural network processing device may be selected based, at least in part, on combination of function values that are computed based, at least in part, on a tensor expressing sample neural network weights.