Neural Network Filter Pruning via Geometric Median Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks, particularly CNN-based systems, require high-performance hardware for real-time processing, and their processing speed is slow on low-power edge devices like CPUs, necessitating improvements in filter pruning or reduction techniques.
Innovation Solution
A method and apparatus for pruning filters in neural networks using geometric median-based clustering, where filters are divided into clusters, and filters are excluded based on geometric median calculations and reduction ratios, optimizing processing speed without significant performance degradation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If filters are pruned using conventional single geometric median-based methods, then processing speed is improved, but inference performance deteriorates and redundant filters are not effectively removed
Solution Approach 1:
The patent segments the filter pruning process into multiple stages: first dividing filters into clusters based on weight distribution, then calculating geometric medians for each cluster, and finally pruning based on cluster-level statistics. This segmentation allows more precise identification of redundant filters while preserving informative ones, thereby maintaining inference performance while improving processing speed.
Solution Approach 2:
The patent applies local quality by calculating geometric medians at the cluster level rather than using a single global geometric median. Each cluster is treated independently with its own geometric median calculation, allowing different pruning thresholds for different filter groups. This enables the model to identify and remove redundant filters in specific local regions while preserving important filters in other regions, thus maintaining overall inference performance.
2Productivity
If more filters are removed to reduce model size, then processing speed on edge devices is improved, but the complexity of determining which filters to remove increases
Solution Approach 1:
The patent performs preliminary action by first clustering filters into groups based on their weight distributions before performing the actual pruning. This pre-organization of filters into clusters with calculated geometric medians creates a structured framework that simplifies the subsequent pruning decision process. The preliminary clustering step reduces the complexity of filter selection by providing clear criteria (cluster-level geometric medians and standard deviations) for identifying redundant filters.
Solution Approach 2:
The patent changes parameters by using cluster-level geometric medians and standard deviations as pruning criteria instead of single global thresholds. This parameter transformation allows for adaptive pruning where different clusters can have different pruning thresholds based on their specific characteristics. The use of statistical parameters (mean, standard deviation) at the cluster level provides objective, data-driven criteria that simplify the filter selection process while enabling aggressive pruning for improved processing speed.
Data Source
AI summary
One or more embodiments relate to a technology for pruning filters or reducing filters based on clustering. According to one or more embodiments, there is provided a method for pruning filters in neural networks, the method including obtaining a convolutional layer having a plurality of filters; generating a plurality of clusters by dividing the plurality of filters; calculating a geometric median for each of the plurality of clusters; and excluding at least one filter from among the plurality of filters based on the geometric median for each of the plurality of clusters.