System and method for pruning a neural network

Abstract

The presently disclosed subject matter provides a system and method for pruning a neural network to optimize complexity based on component separability. A pre-trained neural network and labeled training data associated with a given task are obtained, and multiple sub-tasks corresponding to class pairs are defined. For each component of each layer, separability scores quantifying distinctions between activation distributions are computed and aggregated into separability vectors. A multi-dimensional representation of the separability vectors is generated, and iterative clustering with varying cluster counts is applied. Clustering quality is evaluated using a Mean Simplified Silhouette index indicative of diversity and separability. A number of clusters is selected based on diminishing marginal utility to define a subset of components to retain. Components corresponding to representative cluster points are preserved, and others are pruned, yielding a reduced neural network with maintained task performance and improved computational efficiency.

Description

[0001] SYSTEM AND METHOD FOR PRUNING A NEURAL NETWORK TECHNICAL FIELD

[0002] The present invention relates to the field of neural networks, and more particularly to systems and methods for pruning such networks to obtain lightweight neural networks (NNs).

[0003] BACKGROUND

[0004] Neural networks have revolutionized deep learning, demonstrating remarkable success in tasks such as image classification, object detection, and image segmentation. Despite these achievements, modem neural network architectures, whether convolutional, recurrent, transformer-based, or graph-based typically comprise millions of parameters, rendering them computationally intensive and difficult to deploy on resource-constrained devices. To address this challenge, model compression techniques have emerged as essential tools, enabling reductions in model complexity, computational cost, and memory requirements, thereby facilitating real-world deployment.

[0005] Among the various compression methods, pruning, decomposition, quantization, and knowledge distillation are widely studied. Of these, pruning is particularly effective, as it eliminates redundant components to yield more efficient, sparse networks without substantial performance degradation. The goal is to reduce the number of parameters while preserving accuracy, thereby achieving faster inference and lower storage requirements.

[0006] Pruning techniques can generally be divided into two categories: structured and unstructured pruning. Unstructured pruning removes individual weights from the network based on criteria such as weight magnitude. While this can produce highly sparse models, the resulting irregular structures are often incompatible with modem hardware, which is optimized for dense matrix operations. Consequently, unstructured pruning may not deliver meaningful improvements in computational speed or efficiency. Structured pruning, by contrast, eliminates entire components, such as filters, channels, or layers, thereby preserving regularity and enabling greater hardware acceleration. This approach reduces both parameter count and computational overhead in a manner more aligned with hardware capabilities. Another class of pruning methods leverages network activations, often referred to as activation-based pruning. These approaches remove components based on their activation patterns during the forward pass, which requires access to the training dataset. By analyzing activations, less important components can be identified and pruned without significantly affecting accuracy. While activation-based pruning provides more finegrained decisions, its dependency on the training dataset is a key limitation.

[0007] A persistent challenge across pruning approaches is the need for manual specification of pruning extent. Users are typically required to define the proportion of parameters or components to prune, which often involves iterative trial-and-error to identify an optimal balance between efficiency and accuracy. This process is timeconsuming, computationally expensive, and impractical for large-scale networks or diverse deployment scenarios. The reliance on user input for pruning decisions hinders automation and scalability, limiting the applicability of existing methods in real-world environments.

[0008] Considering the above, there is a need for new systems and methods that automates neural network pruning.

[0009] GENERAL DESCRIPTION

[0010] In accordance with a first aspect of the presently disclosed subject matter, there is provided a system for pruning a neural network, the system comprising a processing circuitry configured to: obtain: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pretrained using labeled training data associated with a given task, and (b) the labeled training data; define, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data; for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculate a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregate the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component; generate a multi-dimensional representation in which each separability vector is represented as a point in a graph; iteratively apply clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters, each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations; calculate, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task; determine a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and, prune, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of said components associated with the representative points.

[0011] In some cases, during generating the multi-dimensional representation of each separability vector, the processing circuitry is configured to perform dimensional reduction on said separability vector of each of said components.

[0012] In some cases, the dimensional reduction is achieved using at least one of: t-SNE, diffusion maps, or u-maps.

[0013] In some cases, the number of clusters corresponding to a point at which marginal utility of additional clusters diminishes is determined by applying a knee-finding algorithm to an MSS-versus-k curve.

[0014] In some cases, at least some of the components corresponding to representative points are selected based on weight magnitude, such that the selected components has the highest absolute weight among any components mapped to its respective cluster, and components other than the selected components are pruned.

[0015] In some cases, applying clustering comprises executing a k-medoids clustering algorithm.

[0016] In some cases, the representative point of each cluster corresponds to the medoid of the cluster.

[0017] In some cases, the processing circuitry is further configured to fine-tune the pruned neural network using at least a portion of the labeled training data.

[0018] In some cases, the at least a portion of said labeled training data involves data which was not used in the executed steps. In some cases, the plurality of components comprises structured components selected from fdters, channels, or neurons, and wherein the pruning removes entire structured components.

[0019] In some cases, calculating said MSS index comprises: for each given clustering corresponding to a given number of clusters, calculating, for each point in said multidimensional representation, a MSS score indicative of a ratio between intra-cluster compactness and inter-cluster separation with respect to said point, and averaging the MSS scores of at least some of said points to obtain a single MSS value representative of clustering quality for said number of cluster.

[0020] In accordance with a second aspect of the presently disclosed subject matter, there is provided a method for pruning a neural network comprising: obtaining: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pre-trained using labeled training data associated with a given task, and (b) the labeled training data; defining, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data; for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculating a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregating the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component; generating a multi-dimensional representation in which each separability vector is represented as a point in a graph; iteratively applying clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters, each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations; calculating, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task; determining a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and, pruning, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of said components associated with the representative points.

[0021] In some cases, during generating the multi-dimensional representation of each separability vector, the method is configured to perform dimensional reduction on said separability vector of each of said components.

[0022] In some cases, the dimensional reduction is achieved using at least one of: t-SNE, diffusion maps, or u-maps.

[0023] In some cases, the number of clusters corresponding to a point at which marginal utility of additional clusters diminishes is determined by applying a knee-finding algorithm to an MSS-versus-k curve.

[0024] In some cases, at least some of the components corresponding to representative points are selected based on weight magnitude, such that the selected components has the highest absolute weight among any components mapped to its respective cluster, and components other than the selected components are pruned.

[0025] In some cases, applying clustering comprises executing a k-medoids clustering algorithm.

[0026] In some cases, the representative point of each cluster corresponds to the medoid of the cluster.

[0027] In some cases, the method is further configured to perform fine-tuning the pruned neural network using at least a portion of the labeled training data.

[0028] In some cases, the at least a portion of said labeled training data involves data which was not used in the executed steps.

[0029] In some cases, the plurality of components comprises structured components selected from filters, channels, or neurons, and wherein the pruning removes entire structured components.

[0030] In some cases, calculating said MSS index comprises: for each given clustering corresponding to a given number of clusters, calculating, for each point in said multidimensional representation, a mean simplified silhouette score indicative of a ratio between intra-cluster compactness and inter-cluster separation with respect to said point, and averaging the mean simplified silhouette scores of at least some of said points to obtain a single MSS value representative of clustering quality for said number of cluster. In accordance with a third aspect of the presently disclosed subject matter, there is provided a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, executable by at least one processor of a computer to perform a method for pruning a neural network comprising: obtaining: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pre-trained using labeled training data associated with a given task, and (b) the labeled training data; defining, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data; for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculating a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregating the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component; generating a multi-dimensional representation in which each separability vector is represented as a point in a graph; iteratively applying clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters, each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations; calculating, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task; determining a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and, pruning, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of said components associated with the representative points. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to understand the presently disclosed subject matter and to see how it may be carried out in practice, the subj ect matter will now be described, by way of non-limiting examples only, with reference to the accompanying drawings, in which:

[0032] Fig. 1 is an exemplary automatic complementary separation pruning algorithm, in accordance with the presently disclosed subject matter;

[0033] Fig. 2 is a two-dimensional representation of a ResNet-56 linear layer component space, where each point represents a component colored by its weight, in accordance with the presently disclosed subject matter;

[0034] Fig. 3 is a bar chart illustrating the effect of polynomial degree on the automatically determined size in the knee / elbow algorithm, and consequently affects the VGG-16 model performance on the CIFAR-10 dataset, in accordance with the presently disclosed subject matter;

[0035] Figs. 4 A and 4B are graphs comparing accuracy under increasing attack strengths (a) for FGSM (4A) and random noise (4B), in accordance with the presently disclosed subject matter;

[0036] Fig. 5 is a block diagram schematically illustrating one example of a system for pruning a neural network, in accordance with the presently disclosed subject matter; and, Figs. 6A and 6B are exemplary flowcharts illustrating an example of a sequence of operations carried out by a system for pruning a neural network, in accordance with the presently disclosed subject matter.

[0037] DETAILED DESCRIPTION

[0038] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the presently disclosed subject matter. However, it will be understood by those skilled in the art that the presently disclosed subject matter may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the presently disclosed subject matter.

[0039] In the drawings and descriptions set forth, identical reference numerals indicate those components that are common to different embodiments or configurations.

[0040] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “obtaining44, “defining”, “calculating44, “aggregating”, “generating”, “applying”, “determining”, “pruning”, “performing”, or the like, include action and / or processes of a computer that manipulate and / or transform data into other data, said data represented as physical quantities, e.g., such as electronic quantities, and / or said data representing the physical objects. The terms “computer”, “processor”, “processing resource”, “processing circuitry”, and “controller” should be expansively construed to cover any kind of electronic device with data processing capabilities, including, by way of non-limiting example, a personal desktop / laptop computer, a server, a computing system, a communication device, a smartphone, a tablet computer, a smart television, a processor (e.g. Digital Signal Processor (DSP), a microcontroller, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), etc.), a group of multiple physical machines sharing performance of various tasks, virtual servers co-residing on a single physical machine, any other electronic computing device, and / or any combination thereof.

[0041] The operations in accordance with the teachings herein may be performed by a computer specially constructed for the desired purposes or by a general-purpose computer specially configured for the desired purpose by a computer program stored in a non-transitory computer readable storage medium. The term "non-transitory" is used herein to exclude transitory, propagating signals, but to otherwise include any volatile or nonvolatile computer memory technology suitable to the application.

[0042] As used herein, the phrase "for example," "such as", "for instance" and variants thereof describe non-limiting embodiments of the presently disclosed subject matter. Reference in the specification to "one case", "some cases", "other cases" or variants thereof means that a particular feature, structure or characteristic described in connection with the embodiment s) is included in at least one embodiment of the presently disclosed subject matter. Thus, the appearance of the phrase "one case", "some cases", "other cases" or variants thereof does not necessarily refer to the same embodiment s).

[0043] It is appreciated that, unless specifically stated otherwise, certain features of the presently disclosed subject matter, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the presently disclosed subject matter, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable sub-combination. In embodiments of the presently disclosed subject matter, fewer, more and / or different stages than those shown in Figs. 6A and 6B may be executed. In embodiments of the presently disclosed subject matter one or more stages illustrated in Figs. 6A and 6B may be executed in a different order and / or one or more groups of stages may be executed simultaneously. Each module in Fig. 5 can be made up of any combination of software, hardware and / or firmware that performs the functions as defined and explained herein. The modules in Fig. 5 may be centralized in one location or dispersed over more than one location. In other embodiments of the presently disclosed subject matter, the system may comprise fewer, more, and / or different modules than those shown in Fig. 5.

[0044] Any reference in the specification to a method should be applied mutatis mutandis to a system capable of executing the method and should be applied mutatis mutandis to a non-transitory computer readable medium that stores instructions that once executed by a computer result in the execution of the method.

[0045] Any reference in the specification to a system should be applied mutatis mutandis to a method that may be executed by the system and should be applied mutatis mutandis to a non-transitory computer readable medium that stores instructions that may be executed by the system.

[0046] Any reference in the specification to a non-transitory computer readable medium should be applied mutatis mutandis to a system capable of executing the instructions stored in the non-transitory computer readable medium and should be applied mutatis mutandis to method that may be executed by a computer that reads the instructions stored in the non-transitory computer readable medium.

[0047] By way of introduction, the presently disclosed subject matter relates to systems and methods for automated neural network pruning, introducing an Automatic Complementary Separation Pruning (ACSP) framework that fully automates the pruning process.

[0048] ACSP may integrate both structured pruning and activation -based pruning, thereby enabling the removal of entire network components, such as channels, neurons, or other types of components, while optionally utilizing activation information to preserve the most critical components. A key aspect of ACSP may include the selection of components based on their complementary capabilities, thereby ensuring diversity of function and reducing redundancy within the pruned network. Unlike conventional pruning approaches that often require manual user input to define a pruning volume, ACSP may be configured to automatically determine the smallest and most diverse subset of components in each layer. In some embodiments, such determination may be achieved, for example, by selecting components that contribute unique, non-overlapping capabilities. In other embodiments, the complementary selection may optionally be performed using a graph-based approach, in which components are selected, for example, from distinct regions within a graph space so as to avoid redundancy. This ensures that each chosen component not only contributes to performance across multiple tasks, but also provides unique separation capabilities. Graph-based complementary selection principles, which have previously been applied in domains such as feature selection and clustering, may thus be extended in the presently disclosed subject matter to neural network pruning.

[0049] The pruning process of the presently disclosed subject matter may be conducted iteratively, on a layer-by-layer basis. For each layer, ACSP may construct a graph space based on activation signals, in which the graph encodes the separation capability of each component relative to class pair activations, rendering the method particularly suited for supervised learning tasks. In certain embodiments, ACSP may enforce complementary selection by identifying components from different regions of the graph space, thereby emphasizing diversity and complementary separation properties.

[0050] In some embodiments, ACSP may utilize a clustering algorithm in conjunction with a knee-finding technique to automatically identify an optimal pruning subset. This automated process may render ACSP efficient, scalable, and suitable for practical deployment on resource-constrained devices.

[0051] It is to be noted that the knee-finding technique mentioned hereinbefore serves as mere example, not intended to limit the scope of the presently disclosed subject matter, as other suitable techniques may likewise be applicable.

[0052] The presently disclosed subject matter may include, without limitation, the following features and advantages:

[0053] • Automated pruning: ACSP may automatically determine an optimal subset of neural network components to prune without requiring manual intervention, thereby overcoming limitations of user-defined pruning volumes and reducing redundancy. • Combined pruning approach: ACSP may integrate structured pruning with activation-based pruning, enabling removal of entire components such as neurons or channels while optionally leveraging activation signals to preserve critical components. Such integration may produce models that are both computationally efficient and hardware-friendly.

[0054] • Complementary selection: ACSP may employ graph-based complementary selection techniques to ensure that the retained components provide diverse and non-overlapping capabilities, thereby balancing efficiency with performance.

[0055] • Practical performance: In some embodiments, ACSP may achieve substantial reductions in computational and memory costs while maintaining, or in certain cases improving, predictive accuracy across multiple neural network architectures and datasets, thereby rendering ACSP practical for real-world deployment.

[0056] Method

[0057] Notation and Definitions

[0058] Before describing specific embodiments of the presently disclosed subject matter, the following notations are defined to facilitate clarity and improve understanding of the disclosed approach.

[0059] In certain embodiments, a neural network may be represented as F(D; W), where D denotes a dataset and W denotes the network weights. The dataset D may comprise (X, T), where X represents input data and Y represents corresponding labels. The labels Y may optionally include C unique classes.

[0060] For a given neural network,

[0061]

[0062] may denote the / -th layer associated with weights Wt. The number of components within layer

[0063]

[0064] such as neurons in a linear layer or channels in a convolutional layer, may be represented as Nt. The activations corresponding to layer may be represented as At.

[0065] In certain embodiments, the set of indices for the components in layer may be denoted as Ii= {1,2,...,Ni}, where Ii,jrepresents the j-th component in layer Li. For a convolutional layer, the activation corresponding to the output of the / -th component, may be an activation map of size p X p for the / -th input sample, where p denotes the spatial dimension. For a linear layer, p may optionally equal 1, such that Ai,j[t] reduces to a scalar value.

[0066] In certain embodiments, a pruning process may be configured to identify, for each layer a subset of the original components

[0067]

[0068] sufficient to preserve, or optionally enhance, the performance of the neural network while reducing its size and computational complexity.

[0069] Method Overview

[0070] Embodiments of the presently disclosed subject matter may utilize an ACSP process to automatically determine the size and composition of the smallest subset of components in each layer of a neural network, without substantially degrading model performance.

[0071] For each layer

[0072]

[0073] a graph space may be constructed that encodes the separability of each component with respect to all possible pairs of classes in the dataset. Each component may be evaluated based on a separation score computed for these class pairs. Such information may be encoded into a vector of size 1 x p x p x (j), representing the position of the component within the graph space corresponding to layer Lj.

[0074] To determine the optimal subset of components for a given layer

[0075]

[0076] a score may be assigned to each candidate subset size within the range

[0077]

[0078] [2,. In certain embodiments, ACSP may employ the principle of complementary selection, wherein components exhibiting diverse and complementary separability capabilities are selected, thereby minimizing redundancy among components with similar separability properties. To implement complementary selection, a clustering algorithm may optionally be applied to identify components from distinct regions of the graph space, ensuring complementary separation capabilities across the retained subset.

[0079] The quality of each subset size may be evaluated using a scoring metric, such as the Mean Simplified Silhouette (MSS) index, which provides a quantitative measure of clustering quality for each tested subset size. Following subset scoring, a knee-finding algorithm may be employed to identify the optimal subset size, and the corresponding subset of components may be selected based on the clustering results. All components of the layer apart from the identified subset may then be pruned. Subsequent to pruning, a fine-tuning process may optionally be conducted on a portion of the dataset to enable the remaining layers to adapt to the pruned architecture and restore optimal performance. The complete algorithmic procedure for this operation is outlined in Fig. 1.

[0080] In some embodiments, the pruning procedure may be applied iteratively across layers of the neural network, beginning with the first hidden layer and proceeding sequentially through to the final hidden layer, while optionally excluding the input and output layers. By employing this iterative approach, the entire model may be pruned in a layer-by-layer manner.

[0081] Graph Space Representation

[0082] Constructing the Graph Space

[0083] In certain embodiments, the objective of this step is to construct, for a given layer L£, a graph space configured to encode the separability of the layer’s components It across all possible class pairs (c, c̃), where 1 ≤ c, c̃ ≤ C. For each component Ii,j, the separability information may be encoded into a vector of size 1 x p x p x (j), representing the position of the component within the graph space of layer L£.

[0084] The process of encoding the separability vector may differ depending on whether the layer is linear or convolutional.

[0085] In certain embodiments corresponding to a linear layer, the process may begin by performing a forward pass of the dataset D through the neural network to extract activations from layer Li. For each input sample x ∈ X, a total of Niactivation values may be obtained, each corresponding to a scalar value associated with a respective neuron Ii,jin layer Li.

[0086] To quantify the separability of a given neuron Ii,jwith respect to a class pair (c, c̃), a statistical distance measure may be employed. In certain embodiments, the Jeffries–Matusita (JM) distance may be computed between the activation values Ai,jcorresponding to samples labeled as class c and those labeled as class c̃ for neuron Ii,j.

[0087] The JM distance between two distributions may be defined as:

[0088] JM(i,j)(c,c̃) = 2(1 - e-B(c,c̃))

[0089]

[0090] where the Bhattacharyya distance B(i,j)(c, c̃) is given by: 1,c 1

[0091] 5(i,7)(c. c)

[0092] 8 a? jc+ <j?j£+2ln

[0093]

[0094] ””LJ—2

[0095] where μi,j,cand σ²i,j,cdenote, respectively, the mean and variance of the activation values Ai,jfor class c, with analogous definitions for class c̃. The JM distance thus provides a separability score for neuron Ii,jbetween the two classes.

[0096] This computation may be repeated for all neurons

[0097]

[0098] within layer

[0099]

[0100] and for all class pairs (c, c̃). The resulting separability values for each component may be encoded into a vector of size 1 × (p × p × (C / 2)), where p = 1 for linear layers. The final matrix representation for layer Limay thus have dimensions Ni× (p × p × (C / 2)), where each row represents the separation capability of a respective neuron across all class pairs.

[0101] In certain embodiments corresponding to convolutional layers, each sample t ∈ X may produce Niactivation maps from layer

[0102]

[0103] where each activation map Ai,j[t] corresponds to a p × p matrix generated by a respective filter. To compute the separability of a channel Ii,jbetween classes (c, c̃), activation maps Ai,jfrom samples labeled as class c and as class c̃ may be compared.

[0104] For each pixel location within these activation maps, the JM distance may be computed between the pixel values of samples belonging to class c and class c. After computing separability values for all pixels, the resulting p × p matrix may be flattened into a vector of size 1 × (p × p). Repeating this computation for all class pairs yields a separability vector of size 1 × (p × p × (C / 2)) for each channel Ii,j.

[0105] Accordingly, the final matrix representation for layer

[0106]

[0107] in the case of convolutional layers may have dimensions Ni× (p × p × (C / 2)), where Nidenotes the number of channels. Each row of this matrix may represent the separation capability of a respective channel across all class pairs.

[0108] It should be noted that the presently disclosed subject matter, described herein, is not limited to any particular separability metric and may be implemented using various alternative measures of distributional divergence or similarity. In certain embodiments, the presently disclosed subject matter may employ one or more separability metrics selected from the group consisting of the Jeffries-Matusita (JM) distance, the Hellinger distance, and the Wasserstein distance, among others.

[0109] Graph Space Evaluation

[0110] In certain embodiments, the presently disclosed subject matter aims to select components exhibiting complementary separation capabilities. At a graphical representation level, this objective corresponds to selecting components distributed across diverse regions of a graph space, thereby ensuring broad coverage of separation characteristics.

[0111] Consider, for illustrative purposes, three components denoted as Ii,j, Ii,k,

[0112]

[0113] and each represented by a respective separation vector within the graph space. When components Ii,jand Ii,kexhibit similar separation capabilities, their corresponding representations are positioned in close proximity within the graph space. Conversely, component Ii,lmay exhibit distinct separation characteristics, resulting in a representation located in a more distant region of the graph space.

[0114] Accordingly, when selecting a subset of components, the presently disclosed subject matter may prioritize combinations comprising components from distinct graph space regions, such as selecting either Ii,jor Ii,kin combination with

[0115]

[0116] Although Ii,jand Ii,kmay individually exhibit higher separation values than Ii,l, their proximity within the graph space renders them redundant with respect to complementary separation capability. Selecting Ii,l, even with a lower separation value, promotes diversity and complementarity among the selected components.

[0117] To facilitate this complementary selection, the presently disclosed subject matter may employ a k-Medoids clustering algorithm to partition the graph space into k clusters. Each cluster may correspond to a set of components exhibiting similar separation characteristics, and the medoid of each cluster may be selected to represent the corresponding subset. The resulting set of medoids thereby forms a subset of components that collectively provide a wide distribution across the graph space, satisfying the principle of complementary selection.

[0118] Since the optimal number of clusters k may vary between 2 and Nt, the presently disclosed subject matter may further include evaluating the clustering quality to verify that the selected components achieve adequate coverage of the graph space. In certain embodiments, such evaluation may be performed using a Mean Silhouette Score (MSS) index or a comparable quality metric configured to measure how effectively the selected medoids represent the desired diversity and distribution across the graph space.

[0119] Mean Simplified Silhouette (MSS) Index

[0120] In certain embodiments, the Mean Simplified Silhouette (MSS) index may be employed to evaluate clustering results in the context of component selection, with an emphasis on diversity, spatial coverage of the graph space, and minimization of redundancy.

[0121] Conventional evaluation indices, such as the Silhouette or Simplified Silhouette, assess the degree to which a point is associated with its assigned cluster relative to its separation from the nearest neighboring cluster. Although effective in standard clustering contexts, these indices focus solely on the nearest cluster and therefore fail to consider the global distribution of clusters within the graph space. Such global consideration is critical for achieving complementary component selection, wherein selected components exhibit diverse and non-overlapping separation characteristics.

[0122] The MSS index addresses this limitation by evaluating the relationship of each point to all clusters rather than only to the nearest cluster. This enables the MSS index to measure not only intra-cluster cohesion but also inter-cluster dispersion, thereby ensuring that selected components are well associated with their respective clusters while also being widely distributed across the entire graph space. As a result, the MSS index promotes the selection of components that are complementary and collectively representative of the separability landscape.

[0123] For each point i, let a(i) denote the distance between point i and the center of its assigned cluster Ch

[0124] a

[0125]

[0126] (i) = d(i,

[0127] Next, let b(i) denote the average distance between point i and the centers of all other clusters Ct, where I A h:

[0128] b(i) = averagel≠hd(i, Cl)

[0129] The MSS score for point i is then defined as:

[0130] a(i)

[0131] mss(i) = 1 - a(i) / b(i) The overall MSS index is computed as the mean of the individual mss(i) scores over all points in the dataset. Higher MSS values indicate better global distribution and complementary clustering quality.

[0132] Automatic Pruning

[0133] Component Size Determination

[0134] To identify a compact subset of components that preserves network performance, the quality of clustering outcomes may be evaluated for each candidate subset size k within the range [2, Ni], where Nirepresents the total number of components in layer Li. For each subset size k, the k-Medoids clustering algorithm may be executed on the graph space, and the resulting clusters may be assessed using the MSS index.

[0135] After computing MSS scores across the candidate subset sizes, the point of diminishing returns, that is, the size beyond which additional components yield minimal improvement in graph-space coverage, may be determined using the K-needle algorithm. The K needle algorithm identifies a “knee point” in the MSS curve, corresponding to the transition between steep improvement and gradual saturation. This knee point indicates the optimal subset size that balances diversity and efficiency while maintaining performance.

[0136] Component Composition Determination

[0137] As described above, the K needle algorithm determines the target subset size k. The corresponding clustering output defines a cluster space comprising k medoids, which collectively represent the principal regions of the graph space and embody the principle of complementary selection.

[0138] However, the clustering process itself does not account for the relative importance of individual components, as reflected by their associated weights. Component weights are significant indicators of contribution to overall model performance: components with higher weight magnitudes generally exert greater influence on the network’s predictive output. Neglecting weight information could therefore lead to a degradation in postpruning accuracy. To mitigate this effect, the selection process may be modified to incorporate weight information. Specifically, for each cluster, the component with the largest weight magnitude may be selected to represent that cluster. In certain embodiments:

[0139] • For convolutional layers, a filter’s importance may be quantified by its Li-norm, providing a measure of overall filter strength.

[0140] • For fully connected (linear) layers, the importance may be represented by the absolute magnitude of the component’s weight.

[0141] This approach ensures that, within each cluster, the selected component not only represents a unique region of the graph space but also exhibits high intrinsic importance based on its weight magnitude. Consequently, the resulting subset preserves both complementary separation characteristics and performance-critical components.

[0142] An illustrative non-limiting example of this process is presented in Fig. 2, showing a two-dimensional projection of a ResNet-56 linear layer’s component space, where each point corresponds to a component colored by its weight. The space comprises seven clusters, with cluster medoids indicated by triangular markers and the highest-weight components in each cluster denoted by rhombus markers. The figure highlights the distinction between medoid-based selection and weight-aware selection following dimensionality reduction of the graph space using t-SNE.

[0143] Experiments

[0144] The following experiments are provided solely for illustrative purposes and should not be construed as limiting the scope of the disclosed subject matter in any way. These experiments are intended to offer individuals skilled in the art a comprehensive understanding of how to make and use the described invention, ensuring full and clear disclosure. They are not, however, exhaustive or restrictive. The scope of the invention is not confined to these experiments, nor should these specific experiments imply that they represent the entirety of testing or applications. Rather, the experiments serve to highlight various aspects of the invention, and other modifications, variations, and adaptations, as would be apparent to those skilled in the art, remain fully within the intended scope of the invention. Setup

[0145] In certain embodiments, the presently disclosed subject matter was evaluated on the CIFAR-10, CIFAR-100, and ImageNet-1K benchmark datasets. The evaluation was performed using several widely adopted convolutional neural network (CNN) architectures, including VGG-16, VGG-19, DenseNet-40, MobileNet-V2, ResNet-50, and ResNet-56.

[0146] Each model was initially trained to its standard baseline accuracy. Following the application of the pruning process described herein, a lightweight fine-tuning phase was conducted immediately after the pruning of each layer. Specifically:

[0147] • For CIFAR-10 and CIFAR-100, fine-tuning was performed for 2 epochs using a random 25% subset of the training data, with an initial learning rate of 0.01 halved after the first epoch.

[0148] • For ImageNet-1K, fine-tuning was performed for 3 epochs using a 25% subset of the dataset, with an initial learning rate of 0.003, halved after the second epoch.

[0149] This limited fine-tuning procedure was sufficient to restore any transient loss in accuracy with negligible computational overhead.

[0150] Evaluation metrics included: (a) Base Accuracy (pre-pruning performance), (b) Pruned Accuracy (post-pruning performance following fine-tuning), and (c) Speed-Up, defined as the ratio of floating-point operations (FLOPs) before and after pruning.

[0151] Unless otherwise specified, the results presented herein were obtained using a second-degree polynomial within the Kneedie algorithm for knee-point detection, combined with weight-based component selection as described in earlier sections.

[0152] CIFAR-10 Results

[0153] The presently disclosed subject matter was evaluated on MobileNet-V2, VGG-16, and ResNet-56 networks.

[0154] MobileNet-V2: The presently disclosed subject matter achieved a post-pruning accuracy of 94.98%, representing an accuracy gain of +0.5% relative to the baseline, while obtaining a 1.93 x speed-up. • VGG-16: The presently disclosed subject matter produced a +0.37% accuracy improvement and the highest inference speed-up (2.59x) among compared techniques, balancing accuracy retention and computational efficiency.

[0155] • ResNet-56: The presently disclosed subject matter yielded an accuracy gain of +0.13% and a 2.15x speed-up, emphasizing computational benefits with minimal loss in predictive performance.

[0156] Model Methad Base Pruned A Are, Speed Up

[0157] Uniform

[0044] 94.47 94.17 -0.30 1.35×

[0158] DCP

[0025] 94.47 94.69 +0.22 1.35×

[0159] DMC

[0045] 94.23 94.49 +0.26 1.66×

[0160] MobileNet-V2 SCOP

[0021] 94.48 94.24 -0.24 1.67×

[0161] A i'o |+<>< 94.45 94.73 +0.33 I.84 K

[0162] SANF p21 94.52 94.9" +0.45 1.85 x

[0163] ACSP <>4.4>; 94.‘»S -0.50 1.03,

[0164] GAL

[0047] 93.96 90.78 -3.18 1.82×

[0165] HRank

[0048] 93.96 93.43 -0.53 2.15×

[0166] GCNP 9339 93.2" +0.17 2.34x

[0167] VGG-16 CHIP

[0050] 93.96 93.86 -0.10 2.38×

[0168] AOFP

[0051] 93.38 93.84 +0.46 2.52×

[0169] APIB

[0052] 93.68 94.08 +0.40 2.50×

[0170] ACSP 93.55 93.92 +0.37 2.59×

[0171] Geometric: |53| 93.59 93.26 -9.33 i."& K

[0172] Polar

[0054] 93.80 93.82 +0.02 1.88×

[0173] CP [9] 92.80 91.80 -1.00 2.00×

[0174] AMC

[0015] 92.80 91.90 -0.90 2.00×

[0175] ResNet-56 HRank

[0048] 93.26 92.17 -1.09 2.00×

[0176] Si l’ 93.59 93.36 -0.23 2. H X

[0177] DepGraph

[0024] 93.53 93.77 +0.24 2.11×

[0178] ResRep

[0056] 93.71 93.71 +0.00 2.12× ACSP 93.69 93.82 +0.13 2.15×

[0179]

[0180] CIFAR-100 Results

[0181] For CIFAR-100, evaluations were performed on VGG-16, VGG-19, and DenseNet-40 architectures.

[0182] • VGG-16: The presently disclosed subject matter achieved the highest postpruning accuracy (74.31%) with an accuracy gain of +0.61%, along with a 2.01 x speed-up.

[0183] • VGG-19: The presently disclosed subject matter achieved 73.90% post-pruning accuracy and a +0.62% gain, maintaining superior accuracy retention across deeper network structures. • DenseNet-40: The presently disclosed subject matter achieved a minimal accuracy reduction of -0.36% while providing a 1.91 x speed-up, demonstrating its efficiency even for densely connected architectures. X Z \ X

[0184] C--; Cf 5 X ""

[0185] n r £ t- 3 S 2-z

[0186] Medel Method Base Pruned A Act* Speed Up

[0187] -rO.42 i.75x Polar (541 73.83 74.25

[0188] v73.54 74. Q9 4-0.55 1.76 x

[0189] VrGrG- itr6 DS(L.pRFC':

[0190] 73.51 73.86 +O.. V- 2. W>

[0191] ACxI1':MK61 2.0i /

[0192] VS.1+ 73.26 73.48 *0.22 1.59x

[0193] 72.56 72.99 4-0.43 1.69 x

[0194] -o. -4 2 Ou - 73.45 73.1!VGG-’» sSp’wi

[0195] ACSP 73 38 73.90 4-9*62 2.11 <

[0196] SOSP J5V1 74. H 73.46

[0197] 74.24 73.17

[0198] Ikmse\vt-40

[0199] 74.64 74*28

[0200] ACSP

[0201]

[0202] ImageNet-1K Results

[0203] For the large-scale ImageNet-1K dataset, evaluations were conducted on MobileNet-V2 and ResNet-50.

[0204] • MobileNet-V2: The presently disclosed subject matter achieved 71.99% accuracy post-pruning, corresponding to a +0.09% gain, and delivered the highest speedup (1.55x) compared to baseline configurations.

[0205] • ResNet-50: The presently disclosed subject matter achieved an accuracy improvement of +0.66% and the largest speed-up (2.25 x), reflecting robust scalability to high-capacity networks. Model Method Base Pruned A Ace. Speed Up

[0206] CC

[0060] 71.88 70.91 -0.97 1.39×

[0207] Rand. Pruning

[0023] 71.91 70.90 -1.01 1.41×

[0208] SANP

[0022] 71.91 72.05 +0.14 1.41× MobileNet-V2 Uniform

[0044] 71.80 69.80 -2.00 1.43×

[0209] AMC

[0015] 71.80 70.80 -1.00 1.43×

[0210] MetaPruning

[0016] 72.00 71.80 -0.20 1.44×

[0211] ACSP ’ ' ’ 71.90 71.99 +0,09 1.55 /

[0212] DECORE

[0061] 76.15 74.58 -1.57 1.73×

[0213] HRank

[0048] 76.15 74.98 -1.17 1.77×

[0214] CHIP

[0050] 76.15 76.30 +0.15 1.81×

[0215] CCP

[0062] 76.15 76.98 +0.83 2.04×

[0216] PaS

[0063] 76.65 76.70 +0.05 2.05×

[0217] SMCP

[0064] 76.20 76.80 +0.60 2.15×

[0218] JMDP

[0065] 76.60 76.00 -0.60 2.15×

[0219] FPGM

[0053] 76.15 75.59 -0.56 2.15×

[0220] ResRep

[0056] 76.15 76.15 +0.00 2.20×

[0221] ACSP 76.32 76.98 +0.66 2.25×

[0222]

[0223] Inference Time Analysis

[0224] Inference times were measured for both batch and single-sample modes. The results were averaged over 100 independent runs using randomized inputs. Prior to measurement, an initial warm-up phase was executed to stabilize GPU performance.

[0225] Input dimensions were set according to dataset standards: 32×32×3 for CIFAR-10 and CIFAR-100, and 224×224×3 for ImageNet-1K. Batch inference used 40 inputs per batch, while single inference used a batch size of 1. Experiments were conducted on a workstation equipped with four NVIDIA Quadro RTX 6000 GPUs (24 GB each).

[0226] Across all datasets, the pruned models demonstrated consistent reductions in inference time. For example:

[0227] On CIFAR-10, MobileNet-V2 achieved a -20.39% batch inference reduction, and VGG-16 achieved a -6.88% single-inference reduction.

[0228] • On CIFAR-100, DenseNet-40 achieved a -5.99% single-inference reduction.

[0229] • On ImageNet-lK, MobileNet-V2 achieved -10.76% batch inference reduction, while ResNet-50 achieved -8.07% single-inference reduction.

[0230] Overall, the average improvement across all datasets was -8.78% for batch inference and -5.56% for single inference. The observed wall-clock time reductions were smaller than theoretical FLOP -based speed-ups, as hardware utilization does not scale linearly with FLOP count. Nonetheless, the results confirm that the pruning method consistently improves both throughput and latency without sacrificing accuracy.

[0231] Analysis of Method Behavior

[0232] The influence of the polynomial degree used in the knee-finding process was examined using VGG-16 on the CIFAR-10 dataset. Fig. 3 illustrates that higher-degree polynomials produce smaller remaining FLOP ratios, indicating more aggressive pruning but reduced model accuracy. Excessive polynomial degree thus results in over-pruning and accuracy degradation.

[0233] Empirical analysis demonstrated that a second-degree polynomial yields the best trade-off between pruning aggressiveness and accuracy retention. Consequently, ACSP adopts degree 2 as the default configuration, providing consistent results across all evaluated architectures and datasets.

[0234] The results confirmed that degree 2 achieves the most favorable balance between model compactness and predictive accuracy. Although the polynomial degree is an external control parameter rather than a core algorithmic component, its consistent behavior enhances the robustness and automation of the pruning framework.

[0235] Additionally, a comparison between regular medoid-based selection and weightbased selection revealed that weight-based selection consistently produced superior accuracy across all conditions. This result supports the use of component-importance weighting in the final pruning stage.

[0236] Robustness Analysis

[0237] In the context of this disclosure, robustness refers to the ability of the pruned model to maintain predictive accuracy under input perturbations, including adversarial and random noise disturbances. This property is particularly relevant to safety-critical or adversarial environments.

[0238] Two perturbation scenarios were evaluated:

[0239] 1. FGSM (Fast Gradient Sign Method) adversarial attack, introducing perturbations defined as:

[0240] δ = ε sign(∇XL(F(X),Y)) where ε controls perturbation magnitude.

[0241] 2. Random noise perturbations, generated from a chosen distribution and scaled by ε.

[0242] The performance of the original (unpruned) and pruned models was compared under identical conditions. Figs. 4A and 4B illustrate accuracy trends across increasing ε values for both perturbation types.

[0243] Across all tested perturbation levels, the pruned models, particularly those selected using weight-based component selection, demonstrated higher accuracy than the unpruned baseline. For instance:

[0244] • Under FGSM perturbations, the weighted pruned model exhibited an average accuracy gain of 5.91%, rising to 7.56% for ε ≥ 0.01.

[0245] • Under random noise perturbations, the overall accuracy gain was 7.16%, reaching 19.51% under strong noise (ε ≥ 0.4).

[0246] These results suggest that the complementary selection principle underlying the pruning process enhances model robustness by reducing representational redundancy, which may otherwise amplify sensitivity to input perturbations.

[0247] Attention is now drawn to Fig. 5, depicting a block diagram schematically illustrating one example of the system for pruning a neural network 500, in accordance with the presently disclosed subject matter.

[0248] In accordance with the presently disclosed subject matter, the system for pruning a neural network 500 (also interchangeably referred to herein as “system 500”) may comprise a network interface 506. The network interface 506 (e.g., a network card, a Wi-Fi client, 3G / 4G client, or any other component), enables system 500 to communicate over a network with external systems and handles inbound and outbound communications from such systems. For example, system 500 may receive, through network interface 506, one or more neural networks to be pruned.

[0249] System 500 may further comprise or be otherwise associated with a data repository 504 (e.g., a database, a storage system, a memory including Read Only Memory - ROM, Random Access Memory - RAM, or any other type of memory, etc.) configured to store data. Some examples of data that may be stored in the data repository 504 include:

[0250] • One or more pre-pruned neural network composed of a plurality of layers;

[0251] • labeled training data associated with one or more given tasks;

[0252] • a plurality of sub-tasks, each corresponding to a pair of distinct classes represented within a labeled training data;

[0253] • One or more separability scores indicative of a degree of distinction between activation distributions of one or more given components across respective classes of one or more sub-tasks;

[0254] • One or more separability vectors representing the separability characteristics of one or more given components;

[0255] • One or more MSS indexes indicative of a level of diversity and separability among the cluster points; etc.

[0256] Data repository 504 may be further configured to enable retrieval and / or update and / or deletion of the stored data. It is to be noted that in some cases, data repository 504 may be distributed, while the system 500 has access to the information stored thereon, e.g., via a wired or wireless network to which system 500 is able to connect (utilizing its network interface 506).

[0257] System 500 further comprises processing circuitry 502. Processing circuitry 502 may be one or more processing units (e.g., central processing units), microprocessors, microcontrollers (e.g., microcontroller units (MCUs)) or any other computing devices or modules, including multiple and / or parallel and / or distributed processing units, which are adapted to independently or cooperatively process data for controlling relevant system 500 resources and for enabling operations related to system’s 500 resources.

[0258] The processing circuitry 502 comprises a pruning module 508, configured to perform a pruning process, as further detailed herein, inter alia with reference to Figs. 6A and 6B.

[0259] Turning to Figs. 6A and 6B there is shown a flowchart illustrating one example of operations carried out by the system for pruning a neural network 500, in accordance with the presently disclosed subject matter.

[0260] Accordingly, the system for pruning a neural network 500 (also interchangeably referred to hereafter as “system 500”) may be configured to perform a pruning process 600, e.g., using pruning module 508. For this purpose, system 500 obtains: (a) a given neural network comprising multiple layers, each including components configured to transform input signals into output signals based on associated weights; the network optionally being pre-trained using labeled training data for a specific task, and (b) the labeled training data itself (block 602).

[0261] System 500 then defines, based on the given task, a plurality of sub-tasks, each corresponding to a pair of distinct classes represented in the labeled training data (block 604)

[0262] For each component of each layer, system 500: (i) for at least some of the subtasks, calculates a separability score indicative of the degree of distinction between activation distributions of that component for the respective classes of the sub-task, and (ii) aggregates the separability scores across said sub-tasks to form a separability vector representing the separability characteristics of the component (block 606).

[0263] System 500 subsequently generates a multi-dimensional representation in which each separability vector is represented as a point in a graph (block 608). In some cases, generating the multi-dimensional representation comprises performing dimensionality reduction on the separability vector of each component. The dimensionality reduction is achieved using at least one of: t-distributed Stochastic Neighbor Embedding (t-SNE), diffusion maps, Uniform Manifold Approximation and Projection (UMAP), or any other suitable dimensionality reduction technique known in the art.

[0264] Next, system 500 iteratively applies clustering to the graph to produce a plurality of clusterings. Each clustering comprises multiple clusters, each including a representative point corresponding to an actual point of the clustering, with the number of clusters varying across iterations (block 610). In some cases, applying the clustering comprises execution of a k-medoids clustering algorithm, in which the representative point of each cluster corresponds to a medoid of the respective cluster.

[0265] For each clustering, system 500 computes a Mean Simplified Silhouette (MSS) index indicative of the level of diversity and separability among the cluster points, thereby evaluating clustering quality for component selection (block 612). In some cases, calculating the MSS index comprises: for each clustering, computing, for each point in the multi-dimensional representation, a simplified silhouette score reflecting the ratio between intra-cluster compactness and inter-cluster separation, and averaging the scores of at least some of the points to obtain a single MSS value representative of clustering quality.

[0266] System 500 then determines a number of clusters corresponding to a point at which the marginal utility of additional clusters diminishes, the determined number defining the subset size of components to be retained in the given layer (block 614). In some cases, the determination of the number of clusters is achieved by applying a kneefinding algorithm to an MSS-versus-k curve.

[0267] It is to be noted that the above serves as a mere example not intended in any way to limit the scope of the presently disclosed subject matter, and that any other suitable algorithm or technique for evaluating clustering quality and determining component subsets may likewise be utilized.

[0268] Finally, system 500 prunes, from at least one layer, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of the components associated with said representative points (block 616). In some cases, at least some of the components corresponding to representative points are selected based on weight magnitude, such that, for each cluster, the component having the highest absolute weight among components mapped to the respective cluster is retained while the remaining components are pruned.

[0269] In some cases, system 500 may be further configured to fine-tune the pruned neural network using at least a portion of the labeled training data, which my include data not used in the previous steps.

[0270] In some cases, the plurality of components comprises structured components selected from filters, channels, or neurons, and the pruning removes entire structured components

[0271] It is to be noted, with reference to Figs. 6A and 6B, that some of the blocks can be integrated into a consolidated block or can be broken down to a few blocks and / or other blocks may be added. It is to be further noted that some of the blocks are optional. It should be also noted that whilst the flow diagram is described also with reference to the system elements that realizes them, this is by no means binding, and the blocks can be performed by elements other than those described herein.

[0272] It is to be understood that the presently disclosed subject matter is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The presently disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Hence, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the present presently disclosed subject matter.

[0273] It will also be understood that the system according to the presently disclosed subject matter can be implemented, at least partly, as a suitably programmed computer. Likewise, the presently disclosed subject matter contemplates a computer program being readable by a computer for executing the disclosed method. The presently disclosed subject matter further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the disclosed method.

Claims

CLAIMS:

1. A system for pruning a neural network, the system comprising a processing circuitry configured to:obtain: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pre-trained using labeled training data associated with a given task, and (b) the labeled training data;define, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data;for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculate a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregate the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component;generate a multi-dimensional representation in which each separability vector is represented as a point in a graph;iteratively apply clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters, each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations;calculate, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task;determine a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and,prune, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determinednumber of clusters, thereby generating a pruned neural network composed of said components associated with the representative points.

2. The system of claim 1, wherein, during generating the multi-dimensional representation of each separability vector, the processing circuitry is configured to perform dimensional reduction on said separability vector of each of said components.

3. The system of claim 2, wherein the dimensional reduction is achieved using at least one of: t-SNE, diffusion maps, or u-maps.

4. The system of claim 1, wherein said number of clusters corresponding to a point at which marginal utility of additional clusters diminishes is determined by applying a knee-finding algorithm to an MSS-versus-k curve.

5. The system of claim 1, wherein at least some of the components corresponding to representative points are selected based on weight magnitude, such that the selected components has the highest absolute weight among any components mapped to its respective cluster, and components other than the selected components are pruned.

6. The system of claim 1, wherein applying clustering comprises executing a k- medoids clustering algorithm.

7. The system of claim 6, wherein the representative point of each cluster corresponds to the medoid of the cluster.

8. The system of claim 1, wherein the processing circuitry is further configured to fine-tune the pruned neural network using at least a portion of the labeled training data.

9. The system of claim 8, wherein said at least a portion of said labeled training data involves data which was not used in the executed steps.

10. The system of claim 1, wherein the plurality of components comprises structured components selected from filters, channels, or neurons, and wherein the pruning removes entire structured components.

11. The system of claim 1, wherein calculating said MSS index comprises: for each given clustering corresponding to a given number of clusters, calculating, for each point in said multi-dimensional representation, a mean simplified silhouette score indicative of a ratio between intra-cluster compactness and inter-cluster separation with respect to said point, and averaging the mean simplified silhouette scores of at least some of said points to obtain a single MSS value representative of clustering quality for said number of cluster.

12. A method for pruning a neural network comprising:obtaining: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pre-trained using labeled training data associated with a given task, and (b) the labeled training data;defining, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data;for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculating a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregating the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component;generating a multi-dimensional representation in which each separability vector is represented as a point in a graph;iteratively applying clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters,each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations;calculating, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task;determining a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and,pruning, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of said components associated with the representative points.

13. The method of claim 12, wherein, during generating the multi-dimensional representation of each separability vector, the method is configured to perform dimensional reduction on said separability vector of each of said components.

14. The method of claim 13, wherein the dimensional reduction is achieved using at least one of: t-SNE, diffusion maps, or u-maps.

15. The method of claim 12, wherein said number of clusters corresponding to a point at which marginal utility of additional clusters diminishes is determined by applying a knee-finding algorithm to an MSS-versus-k curve.

16. The method of claim 12, wherein at least some of the components corresponding to representative points are selected based on weight magnitude, such that the selected components has the highest absolute weight among any components mapped to its respective cluster, and components other than the selected components are pruned.

17. The method of claim 12, wherein applying clustering comprises executing a k- medoids clustering algorithm.

18. The method of claim 17, wherein the representative point of each cluster corresponds to the medoid of the cluster.

19. The method of claim 12, wherein the method is further configured to perform fine-tuning the pruned neural network using at least a portion of the labeled training data.

20. The method of claim 19, wherein said at least a portion of said labeled training data involves data which was not used in the executed steps.

21. The method of claim 12, wherein the plurality of components comprises structured components selected from filters, channels, or neurons, and wherein the pruning removes entire structured components.

22. The method of claim 12, wherein calculating said MSS index comprises: for each given clustering corresponding to a given number of clusters, calculating, for each point in said multi-dimensional representation, a mean simplified silhouette score indicative of a ratio between intra-cluster compactness and inter-cluster separation with respect to said point, and averaging the mean simplified silhouette scores of at least some of said points to obtain a single MSS value representative of clustering quality for said number of cluster.

23. A non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, executable by at least one processor of a computer to perform a method for pruning a neural network comprising:obtaining: (a) a given neural network composed of a plurality of layers, each including a plurality of components configured to transform input signals into output signals according to associated weights, said given neural network being pre-trained using labeled training data associated with a given task, and (b) the labeled training data;defining, based on said given task, a plurality of sub-tasks each corresponding to a pair of distinct classes represented in said labeled training data;for each given component of a given layer of said plurality of layers: (i) for at least some of said plurality of sub-tasks, calculating a separability score indicative of a degree of distinction between distributions of activations of said given component corresponding to the respective classes of said sub-task; and (ii) aggregating the separability scores across said at least some of said sub-tasks to form a separability vector representing the separability characteristics of said given component;generating a multi-dimensional representation in which each separability vector is represented as a point in a graph;iteratively applying clustering to said graph to generate a plurality of clusterings, each clustering of the clusterings comprising a plurality of clusters, each of which including a representative point corresponding to an actual point of the clustering, wherein the number of clusters varies across iterations;calculating, for each clustering, a Mean Simplified Silhouette (MSS) index indicative of a level of diversity and separability among the cluster's points, thereby evaluating clustering quality for a component selection task;determining a number of clusters corresponding to a point at which marginal utility of additional clusters diminishes, the determined number defining a subset size of components to be retained in the given layer; and,pruning, from at least one given layer of said layers, all components except those corresponding to the representative points of the clustering at the determined number of clusters, thereby generating a pruned neural network composed of said components associated with the representative points.

Images