Harmonic Dense CNN Connections for Lower Power and Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
DenseNet's increased computation and memory access efficiency due to layer-input tensor concatenation lead to higher power consumption, necessitating a method to optimize memory access and reduce power consumption.
Innovation Solution
A harmonic densely connecting method for convolutional neural networks that includes input, layer operation, and output steps, utilizing CPU to select and concatenate layer-input tensors based on specific connection rules, reducing connection complexity and optimizing memory access through limited layer-input element selection and bottleneck layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all pre-layer outputs are concatenated in DenseNet, then accuracy is improved, but computation operations and memory access increase leading to higher power consumption
Solution Approach 1:
The patent segments the dense connection by introducing a bottleneck layer that processes only a subset of pre-layer outputs rather than all of them. This divides the full concatenation operation into two stages: first processing a selected subset through the bottleneck layer, then concatenating only the bottleneck output with the current layer input. This segmentation reduces the number of computations while maintaining the beneficial dense connection effects.
Solution Approach 2:
The patent extracts only the essential information from the pre-layer outputs by selecting a subset of them to be processed through the bottleneck layer. Instead of using all pre-layer outputs directly in the concatenation, the method extracts and processes only the most relevant features through the bottleneck layer, reducing computational overhead while preserving accuracy.
2Measurement precision
If channel width of layer-input tensor is increased through concatenation, then accuracy is improved, but memory access efficiency decreases
Solution Approach 1:
The patent segments the memory access pattern by introducing a bottleneck layer that processes a subset of pre-layer outputs. This creates a two-stage memory access pattern: first accessing only the selected subset of pre-layer outputs for bottleneck processing, then accessing the bottleneck output for final concatenation. This segmentation reduces the total volume of data that needs to be accessed and processed in memory.
Solution Approach 2:
The patent extracts only the necessary computational operations by selecting a subset of pre-layer outputs to process through the bottleneck layer. This extraction reduces the channel width of tensors that need to be managed and accessed in memory, thereby improving memory access efficiency while maintaining the accuracy benefits of dense connections.
Data Source
AI summary
A harmonic densely connecting method includes an input step, a plurality of layer operation steps and an output step. The input step is for storing an original input tensor of the block into a memory. Each of the layer operation steps includes a layer-input tensor concatenating step and a convolution operation step. The layer-input tensor concatenating step is for selecting at least one layer-input element tensor of a layer-input set from the memory according to an input connection rule. When a number of the at least one layer-input element tensor is greater than 1, concatenating all of the layer-input element tensors and producing a layer-input tensor. The convolution operation step is for calculating a convolution operation to produce at least one result tensor and then storing the at least one result tensor into the memory. The output step is for outputting a block output.