Parallel CNN Layer Segmentation for Reduced Communication Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large convolutional neural networks (CNNs) are impractically slow due to high computational costs, and existing parallelization methods are inefficient, especially when pooling is used, as they increase communication costs and limit parallelism.
Innovation Solution
A parallel convolutional neural network is implemented across multiple processing nodes, where a subset of layers is interconnected to minimize communication costs while enabling pooling, by selectively communicating activations between nodes, particularly at predetermined intervals or randomly, to balance computation and communication efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the network is partitioned into parallel subnetworks that communicate activations to all other subnetworks at every layer, then parallel computation is achieved, but communication cost increases significantly for CNNs due to their sparse weight matrices
Solution Approach 1:
The patent segments the CNN into distinct computational stages (convolutional layers, pooling layers, fully connected layers) and assigns them to different processing nodes in a pipeline. This segmentation allows each node to specialize in specific operations, reducing the need for frequent inter-node communication while maintaining parallel throughput.
Solution Approach 2:
The patent introduces buffer memory as an intermediary between processing nodes to store intermediate activations. This buffer acts as a mediator that decouples the communication requirements, allowing nodes to process data at different rates without requiring synchronized communication at every layer transition.
2Measurement precision
If pooling is used to increase CNN accuracy, then activation tensor dimensions change, but this restricts the number of computing nodes and increases communication cost in slice-based parallelization
Solution Approach 1:
The patent implements dynamic pipeline configuration that adapts to the presence of pooling layers. When pooling is detected, the system dynamically adjusts the pipeline stages and buffer placements to accommodate the dimension changes, allowing flexible handling of both pooled and non-pooled architectures without fixed limitations on node count.
Solution Approach 2:
The patent changes the parameterization of the parallelization scheme from fixed slice dimensions to flexible stage-based dimensions. This allows the system to adjust the number of nodes and communication patterns based on whether pooling is present, transforming the rigid parameter constraints into adaptive parameters that can accommodate different architectural choices.
3Productivity
If large N is used to allow more computing nodes in slice parallelization, then parallelism increases, but when pooling is used N becomes small which restricts the number of computing nodes
Solution Approach 1:
The patent transitions from parallelizing only across the N dimension (spatial slices) to utilizing multiple dimensions including the depth dimension (u) and the layer dimension. This multi-dimensional parallelization allows the system to maintain high node counts even when N is reduced by pooling, by distributing work across depth slices and pipeline stages rather than relying solely on spatial partitioning.
Data Source
AI summary
A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.