Neural Network Weight Scaling for Hardware Precision Limits

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep neural networks require significant computational power and time for training, especially when dealing with complex data forms, due to the need to optimize highly non-convex functions involving real-valued weight multiplication, which is challenging for existing hardware systems.

Innovation Solution

A method and hardware system that scale connection weight values within a limited range, allowing iterative updates and storage of scaled values, while inversely scaling input values to maintain output integrity, enabling efficient training by utilizing a limited weight range and optimizing precision without manual tuning.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If connection weight values are allowed to span a wide effective range for learning, then the neural network can learn more effectively, but hardware systems cannot store or process such wide ranges of values

Engineering Contradiction:

Improveeffective range of connection weight valuesVSAvoidhardware storage precision

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The connection weight value is segmented into two parts: a scaled component stored in hardware memory with limited precision, and an unscaled component maintained separately. This segmentation allows the effective weight range to exceed hardware storage capabilities while maintaining accuracy through the combination of both parts.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A scaling factor acts as an intermediary between the hardware's limited precision storage and the neural network's requirement for wide effective weight ranges. The scaling factor transforms stored values into the broader effective range needed for learning, bridging the gap between hardware constraints and algorithmic requirements.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If real-valued weight multiplication is used for training neural networks, then learning accuracy is improved, but computational power requirements increase significantly

Engineering Contradiction:

Improvelearning accuracyVSAvoidcomputational power

Core Design Contradiction:

Measurement precisionVSPower

Solution Approach 1:

The representation parameters of connection weights are changed by introducing scaled and unscaled components. This parameter transformation enables the system to maintain high learning accuracy through effective real-valued multiplication while reducing computational burden by storing only scaled values in hardware.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

Connection weights are pre-scaled before being stored in hardware memory. This preliminary scaling action prepares the weights for efficient hardware storage and computation, allowing the neural network to maintain learning accuracy while reducing the computational power needed during training operations.

Inventive Principle:

Principle #10Preliminary action

3Ease of manufacture

If connection weights are scaled to fit hardware limits, then hardware compatibility is achieved, but the range of values available for learning is reduced

Engineering Contradiction:

Improvehardware compatibilityVSAvoidrange of values for learning

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The system dynamically maintains both scaled values (for hardware compatibility) and unscaled values (for learning range). During training operations, the unscaled values are reconstructed by combining scaled hardware values with scaling factors, enabling the system to adapt between hardware constraints and learning requirements.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The scaled connection weight values stored in hardware are nested within a broader effective range achieved through unscaled components. The hardware-stored scaled values form an inner layer, while the unscaled values extend the effective range outward, creating a nested structure that satisfies both hardware limits and learning needs.

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS11138502B2Foiling neuromorphic hardware limitations by reciprocally scaling connection weights and input values to neurons of neural networks

Publication Date: 2021.10.05 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Training a neural network according to a training algorithm, which may iteratively perform the following. Scaled connection weight values are called from a memory. Such values span an initial range within or compatible with the limited range of values allowed by hardware. Based on the values called, effective connection weight values are learned. The values learned span an effective range that differs from the initial range. As learning proceeds, the scaled connection weight values are updated by scaling the values learned, so as for the updated values to span a final range that is within the limited range. The training algorithm instructs to store the updated, scaled values on the memory, in view of a next iterative step.