Latch-Based MAC Unit for NVM Die Neural Network Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep learning accelerators face inefficiencies in performing multiply-accumulate operations, particularly when implemented on non-volatile memory (NVM) dies, which are essential for neural network computations but require more efficient mechanisms for bit manipulation and temporary storage.

Innovation Solution

The implementation of a latch-based multiple-accumulate (MAC) unit on an NVM die, utilizing linear processing without decision branches, allows for high-speed and low-power MAC operations by using a set of latches and a storage register to perform MAC operations directly on the die, enabling efficient neural network processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If MAC operations are performed using conventional processors, then computational flexibility is maintained, but processing speed and energy efficiency deteriorate

Engineering Contradiction:

ImproveMAC operation speedVSAvoidenergy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent replaces conventional processor-based MAC operations with a dedicated latch-based MAC unit that operates directly on the NVM die. This substitution of the computational mechanism achieves high-speed MAC operations (processing 32 MAC operations in parallel per cycle) while reducing energy consumption by eliminating data transfer between separate memory and processing units.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces latches as intermediary elements between the NVM array and the MAC computation unit. These latches serve as temporary storage that enables efficient data manipulation and parallel processing, acting as a bridge that allows the MAC unit to operate directly on NVM data without conventional processor intervention.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If data is transferred between NVM and external processors, then computational versatility is maintained, but processing time and energy consumption increase

Engineering Contradiction:

Improveprocessing throughputVSAvoiddata transfer time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent merges the NVM array and MAC computation unit onto a single die, eliminating the physical separation between storage and processing. This integration allows data to remain on the NVM die throughout the MAC operation, with results written back to the same NVM array, completely eliminating data transfer time to external processors.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The NVM die performs MAC operations autonomously using its own integrated latch-based MAC unit, without requiring external processor intervention. The die serves itself by maintaining data locally and executing computations in-place, eliminating the need for time-consuming data transfers to and from external processing units.

Inventive Principle:

Principle #25Self-service

3Adaptability or versatility

If conventional memory structures are used for MAC operations, then architectural simplicity is maintained, but computational efficiency for neural networks deteriorates

Engineering Contradiction:

Improveneural network processing capabilityVSAvoidcomputational architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the computational architecture into dedicated functional units: NVM arrays for data storage, latches for temporary storage and bit manipulation, and latch-based MAC units for computation. This segmentation allows each component to be optimized for its specific function, achieving high neural network processing efficiency while maintaining a relatively simple overall structure.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The latch-based MAC unit serves multiple functions: it performs MAC operations, manipulates bits using latch logic, and writes results back to NVM. This multi-functionality allows the same hardware structure to handle various neural network operations (multiply-accumulate, bitwise operations, accumulation) without requiring separate specialized units for each operation type.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12032959B2Non-volatile memory die with latch-based multiply-accumulate components

Publication Date: 2024.07.09 SANDISK TECHNOLOGIES LLC

AI summary

Latch-based multiply-accumulate (MAC) operations implemented on the die of a non-volatile memory (NVM) array are disclosed. The exemplary latch-based MAC procedures described herein are linear procedures that do not require logic branches. In one example, the MAC operation uses a set of linear MAC stages, wherein each linear stage processes MAC operations corresponding to one bit of a first multi-bit multiplicand being multiplied against a second multi-bit multiplicand. Examples are provided wherein the MAC procedures are performed as part of a neural network feedforward procedure where the first multiplicand is a synaptic weight and the second multiplicand is an activation value. Multiple plane and multiple die NVM array implementations are also described for massive parallel processing.