CIM Array Depthwise Separable Convolution Kernel Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional compute-in-memory (CIM) arrays are unable to efficiently process depthwise separable convolutional neural networks without additional hardware, such as digital multiply-and-accumulate circuits (DMACs), which increases power consumption, space requirements, and processing latency.
Innovation Solution
The implementation of a CIM architecture that stores depthwise and fused convolution kernels within the CIM array, allowing for in-memory processing of pre-activations to generate depthwise and fused outputs without the need for external DMACs, by fusing weights for both phases of depthwise and pointwise convolutions and using fused weights to reduce data movement and power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If additional digital multiply-and-accumulate circuits (DMACs) are added to enable depthwise separable convolution processing in CIM, then processing capability is improved, but power consumption and space requirements increase
Solution Approach 1:
The patent combines the depthwise convolution kernel and pointwise convolution kernel into a single unified kernel structure that can be stored in the CIM array. This merging eliminates the need for separate DMAC circuits to handle the two distinct convolution operations, as the unified kernel enables both depthwise and pointwise convolutions to be performed through a single in-memory computation pass, thereby reducing power consumption while maintaining processing capability.
Solution Approach 2:
The unified kernel structure serves multiple functions: it performs both depthwise convolution (when applied to individual channels) and pointwise convolution (when applied across channels) operations. This multi-functionality allows the CIM array to handle depthwise separable convolutions without requiring specialized additional hardware circuits, thus avoiding the power and space overhead of separate DMACs.
2Adaptability or versatility
If additional digital multiply-and-accumulate circuits (DMACs) are added to enable depthwise separable convolution processing in CIM, then processing capability is improved, but space requirements increase
Solution Approach 1:
The patent merges the depthwise and pointwise convolution kernels into a single unified kernel that is stored in the CIM array's memory structure. This consolidation eliminates the need for separate DMAC circuitry that would otherwise be required to implement depthwise separable convolutions, thereby reducing the hardware footprint and space requirements while preserving full processing capability.
Solution Approach 2:
The unified kernel structure provides universal functionality by enabling both depthwise and pointwise convolution operations through a single computational mechanism. This eliminates the need for additional specialized hardware circuits, allowing the system to achieve enhanced adaptability for depthwise separable convolutions without increasing space requirements.
3Adaptability or versatility
If data is moved to and from external DMACs for processing, then complex model architecture processing is improved, but processing latency increases
Solution Approach 1:
The patent combines the depthwise and pointwise convolution operations into a single unified computational process that executes entirely within the CIM array. By merging these operations and eliminating the need for external DMACs, the system performs all necessary computations in-memory without requiring data to be transferred out and back into the memory array, thus reducing processing latency while maintaining the ability to process complex depthwise separable convolutional models.
4Adaptability or versatility
If data is moved to and from external DMACs for processing, then complex model architecture processing is improved, but power usage increases
Solution Approach 1:
The patent merges the depthwise and pointwise convolution kernels into a unified kernel structure that enables both operations to be performed within the CIM array without external DMAC assistance. This consolidation eliminates the energy-consuming data movement operations between memory and external circuits, significantly reducing power usage while preserving the capability to process complex depthwise separable convolutional neural networks.
Data Source
AI summary
Certain aspects of the present disclosure provide a method, including: storing a depthwise convolution kernel in a first one or more columns of a CIM array; storing a fused convolution kernel in a second one or more columns of the CIM array; storing pre-activations in one or more input data buffers associated with a plurality of rows of the CIM array; processing the pre-activations with the depthwise convolution kernel in order to generate depthwise output; modifying one or more of the pre-activations based on the depthwise output to generate modified pre-activations; and processing the modified pre-activations with the fused convolution kernel to generate fused output.