Method, system and computer program for developing a neural network model

Abstract

The present disclosure relates to a computer-implemented method for developing a neural network model, the method providing the ability to adapt the size of the neural network during training. The method comprises setting up a neural network including a given number of layers l, wherein, during training, each neuron's activation at layer l is multiplied by a pre-defined function f αl (i) for each layer l, where i is the position of the neuron and αl is a learnable parameter for layer l. Furthermore, the method comprises using a training dataset for training the neural network, wherein training includes at each iteration: adaptively selecting the width of the layers l of the neural network based on a pre-defined threshold value of function f αl(i); computing the current value of the neural network's loss function to be minimized, and computing gradients of the neural network's loss function and updating the parameters of the neural network including parameter αl. After training, the method outputs the trained neural network model for making predictions on unseen data. Applications of the present disclosure include, but are not limited to, optimized decision making for use cases in drug development, material informatics, data security, cyberthreat intelligence, data extraction, digital transformation, and medical / healthcare.

Description

[0001] METHOD, SYSTEM AND COMPUTER PROGRAM FOR DEVELOPING A NEURAL NETWORK MODEL

[0002] The present application claims priority to EP 25 152582, filed January 17, 2025, which is hereby incorporated by reference in its entirety herein.

[0003] The present invention relates to a method, system, and computer program for developing a neural network model.

[0004] Neural networks (NNs) are one of the fundamental building blocks of most machinelearning architectures. NNs can be applied to a variety of different data types, including tabular data (Multi-Layer Perceptron, or MLP), image data (Convolutional Neural Networks, or CNNs), temporal data (Recurrent Neural Networks, or RNNs), and graph data (Deep Graph Networks, or DGNs). NNs are constructed from interconnected artificial neurons, which are typically aggregated into layers. Training a NN typically requires making a-priori decisions about some architectural choices (called hyper-parameters) such as the number of hidden layers, the nature of the non-linear activation functions, the initialization of neuron weights, and the number of neurons for each hidden layer.

[0005] The universal approximation theorem (informally) states that an MLP with one hidden layer can approximate any continuous function, provided a sufficient amount of neurons is used. The question of how many neurons are “enough” is often left to the practitioner, who tests many different configurations until the performances of the NN are satisfactory. This approach, however, has some limitations:

[0006] Trying different hyper-parameter configurations, typically through a grid or random search scheme, requires running many training runs. If, for example, one tests MLPs with up to L possible depth and D width configurations for each layer, the number of experiments to run is at least proportional to DAL. As there are usually many other hyper-parameters, practitioners often assign the same size to all layers, which reduces the number of potential configurations to at least D*L.Furthermore, the practitioner may have a wrong bias about the “best” number of neurons for each layer, which may lead to sub-optimal results. The best number of neurons for each layer depends on the task at hand, and it cannot be anticipated in advance.

[0007] There are some notable exceptions to this. For instance, the cascade correlation algorithm (for reference, see Fahlman, Scott, and Christian Lebiere. "The cascadecorrelation learning architecture." Advances in neural information processing systems 2 (1989)) adds a new neuron that maximizes the correlation between its output and the residual error of the neural network. Another example is the selfexpanding NN (for reference, see Mitchell, Rupert, Martin Mundt, and Kristian Kersting. "Self expanding neural networks." arXiv preprint arXiv:2307.04526 (2023)), which relies on natural-gradient heuristics to decide when to expand the size of a hidden layer.

[0008] In recent years, there has been a considerable interest around Large Language Models (LLMs), which consist of very deep neural network architectures trained for weeks on large clusters of machines. Due to the impractical costs of running such networks on low-power devices such as phones, researchers have started to reduce the size of these networks through distillation, which roughly means training a smaller network to imitate the large one, and through pruning, which roughly means setting the values of the neural network weights to zero if they do not belong to a given interval. For further details in this context, please refer to Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv: 1510.00149 (2015). However, distillation is typically costly, since it requires training of another network (which again requires hyper-parameter selection). Furthermore, pruning does not guarantee reduced computational costs, although some efforts have been made to design pruning-aware hardware (for reference, see Mishra, Asit, et al. "Accelerating sparse deep neural networks." arXiv preprint arXiv 2104.08378 (2021)).

[0009] It is an objective of the present invention to improve and further develop a method, a system and a computer program for developing a fully differentiable neuralnetwork model by providing a straightforward concept to control the compromise between performance and computational costs with no overhead.

[0010] This objective is addressed by the subject-matter of the independent claims.

[0011] In accordance with the invention, the aforementioned object is accomplished by a computer-implemented method for developing a neural network model. The method comprises setting up a neural network including a given number of layers ■£, wherein, during training, each neuron’s activation at layer ■£ is multiplied by a pre-defined function fa^i) for each layer where i is the position of the neuron and a{is a learnable parameter for layer ■£. Further, the method comprises using a training dataset for training the neural network, wherein training includes at each iteration: adaptively selecting the width of the layers I of the neural network based on a predefined threshold value of function

[0012]

[0013] computing the current value of the neural network’s loss function to be minimized, and computing gradients of the neural network’s loss function and updating the parameters of the neural network including parameter a{. After training, the method outputs the trained neural network model for making predictions on unseen data.

[0014] In various embodiments, the proposed concept according to the present disclosure provides a machine learning methodology to adapt the size of neural networks during or after training. According to the proposed concept, a function fa^i) is introduced for each layer wherein each neuron’s activation at layer ■£ is multiplied by this function fa i) during training. A pre-defined, e.g. user-chosen, threshold value of function fa^i) is used to adaptively select the width of the layers I of the neural network. The outcome of the training is an adaptive width neural network model that is trained to make predictions on unseen data, e.g. to predict certain properties of interest for a given sample.

[0015] The proposed concept provides a general machine learning method to easily adapt the size of neural networks during and post training. More specifically, the proposed concept provides an easy-to-use technique to learn an unbounded width of a neural network’s layer during training (“unbounded” width here means without imposing upper bounds). The technique does not rely on alternate optimization nor hand-crafted gradient heuristics; rather, it jointly optimizes the width and the parameters of each layer via simple backpropagation.

[0016] One of the key advantages of dynamically learning the width of a neural network lies in the substantial time savings compared to traditional model selection processes. In model selection, tuning the width of each layer often requires an exhaustive search across multiple configurations C for each layer. Since the number of possible configurations grows exponentially with the number of layers L (in fact, this is CAL), it can quickly become computationally prohibitive to try even a few configuration per layer on deep networks. By dynamically adjusting the width during training, the concept proposed herein eliminates the need for this costly search, enabling the model to self-optimize and focus resources on learning effectively with a single run. This streamlined approach drastically reduces experimentation time while maintaining or improving performance.

[0017] Furthermore, in contrast to the state of the art on graph machine learning, the proposed concept does not require fixing a maximum number of neurons for each layer a priori and the technique is fully differentiable. In addition, it should also be noted that the introduced modification with regard to the neural activations does not alter the ability to parallelize computation.

[0018] Applications of the concept proposed herein include, but are not limited to, optimized decision making for use cases in drug development, material informatics, data security, cyberthreat intelligence, data extraction, digital transformation, and medical / healthcare.

[0019] According to an example, the function fa^i) is constructed in such a way that it imposes a soft-ordering of an importance of the neurons at each layer I. For instance, this can be accomplished by any arbitrarily chosen monotonically decreasing function. Together with the neural network’s parameters, such monotonically decreasing functions computing the importance of neurons at each neural network layer can also be learned. These functions allow to dynamically decide fully gradient-driven how many neurons are needed at each layer during training.According to an example, the function

[0020]

[0021] is a distribution. This proves to be particularly useful in terms of defining the threshold value of the function

[0022]

[0023] Generally, the threshold value may be a user-chosen threshold value that is decided prior to training the neural network. For instance, the threshold value may be defined as the quantile function of the distribution evaluated at a chosen value q. Consequently, the width of a layer I of the neural network will be selected as the quantile function of the distribution.

[0024] According to an example, adaptively selecting the width of a layer I of the neural network based on a pre-defined threshold value of function

[0025]

[0026] may comprise a step of determining that the current number of neurons in the layer I is higher than the threshold value, and a step of removing the neurons corresponding to the smallest values of fa#(i) from the layer I until the number of neurons matches the threshold value. Accordingly, the layer’s I size shrinks according to the current gradient obtained through the adaptation of the parametric function fai).

[0027] On the other hand, according to an example, adaptively selecting the width of a layer I of the neural network based on a pre-defined threshold value of function

[0028]

[0029] may comprise a step of determining that the current number of neurons in the layer I is lower than the threshold value, and a step of adding neurons to the layer I until the number of neurons matches the threshold value. Accordingly, the layer’s I size grows according to the current gradient obtained through the adaptation of the parametric function fai).

[0030] According to an example, removing one or more neurons from a layer I as well as adding one or more neurons to a layer I may be performed by changing the output dimension of weighting matrix

[0031]

[0032] and the input dimension of weighting matrix

[0033]

[0034] appropriately. In case of adding neurons, the new weights in the weighting matrices need to be initialised.

[0035] According to an example, the neural network model may be adapted to specific hardware requirements by means of a post training truncation of the neural network model. For instance, the soft-ordering on the neurons importance mentioned abovewould allow to perform straightforward truncation of the network model after training. In particular, when the neurons’ importance functions are monotonically decreasing, the resulting network model can be truncated after training simply by removing the last rows / columns of the weight matrices corresponding to the least important neurons. Consequently, the concept allows to straightforwardly trade-off computational requirements with performance, in a principled way and at zero extra cost, by removing as many neurons as needed as long as they are the least important according to the above functions.

[0036] According to an example, the proposed concept may be applied to a Large Language Model, LLM. This means that the neural network initially set up is an LLM and that the training dataset includes training samples to train the LLM. Using the proposed concept may lead to a hardware-customized LLM, wherein various advantages compared to conventional LLMs may be achieved. State of the art LLMs require enormous resources to be trained, in the order of millions of dollars and thousands of GPUs, and they tend to be so large that the only viable business model is Software as a Service (SaaS). Moreover, there are privacy concerns about storing user conversational data in a company’s servers, especially because this data is often used to update the LLM itself. Hence, companies are starting to reduce the size of LLMs by training smaller models via distillation techniques or by applying quantization and pruning approaches. None of these techniques can control the trade-off between performance and hardware requirements in a principled way. Therefore, a lot of time might be spent retraining a distilled model or trying different pruning techniques.

[0037] According to an example, the concepts proposed herein, including the training step of adaptively selecting the width of the layers I of the neural network based on a pre-defined threshold value of function fai), can be applied to the MLPs inside Transformer layers of LLMs to learn the right number of neurons. This might already result in a reduction of size compared to the solution found by hyper-parameter tuning. Afterwards, depending on the client’s hardware requirements, the number of neurons of each Transformer layer can be carefully reduced, trading off performance for computational costs, in a principled way and without any re-training required. The truncated LLM may then be deployed on the client’s system, forinstance a mobile phone, to be used. Accordingly, using the proposed concept in connection with LLMs will output an LLM that can be deployed, in a privacypreserving way, on a user system and can be used to answer the user’s queries for a complex task. A system that preserves privacy has the advantage to be more willingly accepted by people.

[0038] According to an example, the proposed concept may be applied to a machinelearning interatomic potentials, MLIP, architecture. This means that the neural network initially set up is an MLIP architecture and that the training dataset includes training samples related to a molecular system to simulate using the MLIP architecture. Using the proposed concept may lead to a cost-effective training of MLIPs for simulation of molecular systems, for instance with application to drug design, material science or the like. Using the proposed concept in connection with MLIPs will output an MLIP that can be used to simulate the system of interest. The simulation of the system allows to determine its properties and therefore take decisions about the suitability of the system for specific needs, e.g. material science or drug design needs.

[0039] Generally, MLIPs allow to achieve a good trade-off between fast and accurate molecular simulations. However, the training of MLIPs faces a significant challenge due to data scarcity in real-world scenarios, necessitating the use of Active Learning (AL). AL relies on querying an oracle for the ground truth (e.g., Density Functional Theory), which is computationally expensive to run for large systems.

[0040] To achieve optimal performance, it is crucial to select an appropriate MLIP architecture, such as determining the ideal number of neurons in the model. Common methods for this selection, including grid search, random search, or Bayesian optimization, require multiple training runs to explore the architecture space. Each of these runs would need to employ AL, leading to repeated oracle queries and compounding the associated costs. This creates a substantial bottleneck, as the process becomes both time and resource consuming.

[0041] According to an example, the concepts proposed herein, including the training step of adaptively selecting the width of the layers I of the neural network based on apre-defined threshold value of function fai), can be applied to train an MLIP in a single training process while simultaneously determining an optimal width of each hidden layer for the task of predicting energies and forces. By dynamically learning the appropriate size of the network during training, the approach reduces the need for extensive hyperparameter searches, such as grid search or Bayesian optimization, which typically require numerous training runs to evaluate different architectures.

[0042] This leads to a significant reduction in costs, as the method avoids repeated interactions with an expensive ground-truth oracle and reduces the computational overhead of retraining. Experimental results indicate that the system is stable in terms of the size learned with respect to different choices of other hyper-parameters, which makes training an MLIP suitable in resource-constrained environments like the one above, ensuring both accuracy and computational feasibility.

[0043] Generally, the proposed concept is a general, core ML technique that can be applied to neural networks of different nature, which work on tabular, image, temporal, and graph data. Accordingly, in various examples, the neural network can be a Multi-Layer Perceptron, MLP, a Convolutional Neural Network, CNN, a Recurrent Neural Network, RNN, or a Deep Graph Network, DGN. In case of CNNs, RNNs and DGNs, the proposed concept may not be generically applied to these networks; rather, these networks contain MLPs, whose layers’ width can be adaptively learned during training, as explained above.

[0044] An aspect of the present disclosure relates to a system comprising one or more processors and one or more storage devices. The system is configured to perform the above method.

[0045] Another aspect of the present disclosure relates to a computer program comprising instructions which, when the program may be executed by a computer, cause the computer to carry out the above method.

[0046] Another aspect of the present disclosure relates to a non-transitory, computer-readable medium comprising a program code that, when the program code may beexecuted on a processor, a computer, or a programmable hardware component, causes the processor, computer, or programmable hardware component to perform the above method.

[0047] There are several ways how to design and further develop the teaching of the present invention in an advantageous way. To this end, it is to be referred to the dependent claims on the one hand and to the following explanation of preferred embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the preferred embodiments of the invention by the aid of the figure, generally preferred embodiments and further developments of the teaching will be explained. In the drawing

[0048] Fig. 1 is a diagram schematically illustrating conventional neural network training vs. an example of the proposed concept for neural network width adaptation,

[0049] Fig. 2 is a flow chart of an example of a computer-implemented method for developing a fully differentiable neural network model,

[0050] Fig. 3 is a block diagram of an example of a system for developing a fully differentiable neural network model,

[0051] Fig. 4 is a diagram illustrating a comparison between a conventional MLP (left) and an example of an adaptive width network for an MLP (right) according to the proposed concept, and

[0052] Fig. 5 is a diagram illustrating a process of deciding on an addition of new neurons to a particular layer of a neural network with width adaptation according to the proposed concept.

[0053] Throughout the present disclosure, like reference numbers denote like or at least substantially similar elements.The vast majority of neural networks today, whether shallow or deep, rely on a fixed choice of the number of neurons in their hidden layers. This property is typically treated as an architectural design choice, one of the many hyper-parameters that have to be carefully tuned whenever a new task is approached. The tuning process is known by many names, such as model selection, hyper-parameter tuning, and cross-validation, and it is associated with non-negligible costs: Different architectural configurations are to be trained until one that performs best on a validation set is selected. The configurations’ space grows exponentially in the number of layers, so practitioners often resort to shortcuts such as picking a specific number of hidden units for all layers, which greatly reduces the search space together with the chances of selecting a better architecture for the task. Other techniques to explore the hyper-parameter space include constructive approaches, which alternate optimization of different objective functions to optimize the architecture and network’s parameters, natural gradient-based heuristics that dynamically modify the network, bi-level optimization, and neural architecture search, which often requires separate training runs for each configuration.

[0054] The hyper-parameters’ space exploration problem is exacerbated by the steep increase in size of recent neural architectures for language and vision, for example, where parameters are in the order of billions to (supposedly) accommodate for a huge dataset. Training models so large requires an amount of time, compute power, and energy that currently makes it unfeasible for most research and industrial institutions to perform thorough model selection and find good width parameters; the commonly accepted compromise is to stick to previously successful hyperparameter choices.

[0055] To address the above-mentioned issues, the present disclosure provides a method for developing a fully differentiable neural network that adapts the width of each hidden layer during training. According to embodiments, this is done in such a way that the neural network can later (i.e. post training) be truncated to fit precise hardware requirements.

[0056] Fig. 1 schematically illustrates an example of the proposed concept. Conventionally, given a training dataset 1, a training process 100 is carried out on the trainingdataset 1 to obtain a trained neural network 2, as shown in Fig. 1 a). According to the proposed method, the training process 100 is modified, basically by including a width adaptation process 110 and a process S120 of running inference with neuron’s importance, as shown in Fig. 1 b). At the end of training 100, a trained neural network 3 with task-dependent layers’ sizes is produced. Accordingly, the network used for training is sometimes denoted herein as Adaptive Width Network (AWN) and, correspondingly, the entire process including the width adaptation process 110 and the process 120 of running inference with neuron’s importance is sometimes denoted herein as Adaptive Width Network (AWN) generation process.

[0057] The proposed concept combines two components: One to adaptively select the number of neurons for each layer of the neural network (schematically illustrated at 110), and another that modifies the conventional inference step to account for neurons’ importance at each layer (schematically illustrated at 120). Specifically, in the example illustrated in Fig. 1, training is performed, based on the given training dataset 1 , by alternating step 110, in which the width of each neural network’s layer is computed, step 120 in which inference with neuron’s importance is run, a computation of the loss and an updating of the parameters of the model (including those responsible for the width adaptation) using backpropagation.

[0058] In the following, each of these steps is described in more detail. It is noted that, for the ease of understanding, the following discussion is centred around the MLP as it is used in many complex neural network architectures, such as CNNs and Transformers. However, further below it is explained in detail how the concept can be easily extended to the domain of images, sequences, and graphs.

[0059] Fig. 2 is a flow chart of an example of a computer-implemented method for developing a fully differentiable neural network model that adapts the width of each hidden layer during training and can be later truncated to fit precise hardware requirements. The method comprises providing 210 a dataset that consists of training samples and their respective target values to predict. With regard to the type of training samples and the respective target values, the present disclosure is in principle not subject to any restrictions. Furthermore, the method comprises choosing 220 a neural network, a family of functions fa^i) for each layer and acorresponding threshold (e.g., in the case of functions that are distributions, the quantile function of the distribution evaluated at 0.90).

[0060] Based on these provisions, the method comprises training 230 the neural network in accordance with the proposed concept, i.e. using the AWN generation process as explained above in connection with Fig. 1 and backpropagation to optimize the loss function. Training 230 includes, at each iteration, adaptively selecting 231 the width of the network based on the threshold chosen at 220, performing 232 inference to produce output predictions, computing 233 the current value of the loss function to be minimized, and computing 234 gradients of the loss function and updating the parameters of the neural network and of the functions

[0061]

[0062] By learning the number of neurons for each layer, the proposed concept accomplishes a reduction of the combinatorial time required to perform hyper-parameter tuning and finding a size for each layer that maximizes the performance.

[0063] After training 230, the method comprises outputting 240 a neural network model that predicts the properties of interest for a given sample.

[0064] Optionally, if hardware requirements are not satisfied already, the method comprises truncating 250 the neural network such that it complies with hardware resources. By imposing a soft-ordering on the importance of each neuron during training, it is possible to easily truncate the network post-training as it is sufficient to remove the least important neurons of each layer, which will produce a gradual reduction of the model’s performances.

[0065] Fig. 3 shows a schematic diagram of a corresponding system 10, e.g., a computer system, for developing a fully differentiable neural network model, with the system being configured to perform the method of Fig. 2. The system 10 comprises an optional interface 12 and one or more processors 14 to perform the method of Fig. 2. Features introduced in connection with the method of Fig. 2 may likewise be performed by the system 10 of Fig. 3. For example, the system 10 may comprise machine-readable instructions for providing the functionality of the method, with the one or more processors 14 executing the machine-readable instructions to perform the method of Fig. 2. The one or more processors 14 are coupled with the interface12 and with memory and / or storage, such as one or more storage devices 16. For example, the one or more processors 14 may be configured to provide the functionality of the system 10, e.g., in conjunction with the interface 12 (for exchanging information) and / or with the memory / storage 16 (for storing information).

[0066] For example, the interface 12 may include or correspond to a network interface circuitry and / or a device interface circuitry configured to be communicatively coupled to one or more other devices, such as the one or more processors 14. For example, the interface 12 may include a transmitter, a receiver, or a combination thereof (e.g., a transceiver), and may enable wired communication, wireless communication, or a combination thereof. For example, the one or more processors 14 may include or correspond to a digital signal processor circuitry (DSP), a graphical processing unit (GPU), and / or a central processing unit (CPU). The one or more processors 14 may be coupled to the memory / storage circuitry(s) 16. The memory / storage circuitry 16 may include instructions (e.g., executable instructions), such as computer-readable instructions or processor circuitry-readable instructions. The instructions may include one or more instructions that are executable by a computer, such as by the one or more processors 14. For example, the memory / storage circuitry 16 may include or correspond to volatile or nonvolatile storage circuitry, such as Random Access Memory (RAM), magnetic disks, optical disks, or flash memory devices. The memory / storage circuitry 16 may include both removable and non-removable memory devices.

[0067] Fig. 4 a) illustrates the setup of a conventional MLP as well as the architectural changes according to an example of the concepts proposed herein (Fig. 4 b)). The conventional MLP shown in Fig. 4 a) is composed of a series of a number of L fully connected layers, each parametrized by a weight matrix W whose size depends on a previously user-chosen hyper-parameter. The number of input features is F and the number of output dimensions is 0. The symbol £ and a denote the aggregation of incoming neurons activations and the subsequent application of a non-linear activation function, respectively.Fig. 4 b) illustrates an AWN version of the MLP shown Fig. 4 a), i.e. a neural network modified according to the concept of the present disclosure. As shown, the training process of the conventional MLP is augmented by multiplying each neuron’s activation by a learned scalar

[0068]

[0069] where i is the position of the neuron and a{is a layer-wise learnable parameter for layer ■£.

[0070] More specifically, the AWN version of the MLP relies on arbitrarily chosen, monotonically decreasing functions fa#(i) for each layer ■£, which are parametrized by learnable ca{and define the importance of neuron i at that specific layer. Accordingly, in contrast to a conventional MLP, in the training process proposed herein the i-th neuron’s activation at layer ■£ is multiplied by fai), encouraging a soft-ordering of importance among neurons.

[0071] Like in the conventional MLP, the output of the AWN version of the MLP is a vector y that is used to compute a loss function. The parameters of the network are optimized to minimize the loss function over the dataset as in classical machine learning via backpropagation. However, in contrast to the conventional MLP, the optimization process also includes the newly introduced parameters ca{. In this context it is important to note that for the AWN version of the MLP, the user does not specify a fixed number of neurons for each layer as in the conventional MLP, rather it proposes an initial value for ca{which, as described in detail below, is directly related to the initial size of layer ■£.

[0072] Fig. 5 is a diagram illustrating a decision process for determining when to add new neurons to a particular layer of a neural network with width adaptation according to the proposed concept. According to the illustrated example, the decision process includes a check whether, according to recent variations of OL{, the number of neurons of layer ■£ needs to be adapted. The check is performed at training time, and before inference starts on a new set of samples. It should be noted that in the example, there is no upper bound on the final size of each layer.

[0073] Specifically, Fig. 5 shows the logical process to adapt the size of a hidden layer in the AWN network. The importance of each neuron is controlled by the learnable functions Accordingly, at first, a user-chosen threshold for the functions fa^i)is decided prior to training. If, for instance, the function

[0074]

[0075] is a distribution, this threshold can simply be the quantile function of the distribution evaluated at a chosen value q, representing the value before which q percent of the whole probability mass is produced.

[0076] Next, the threshold value is checked against the current number of neurons in the respective layer. If both are equal, i.e. the threshold value is the same as the number of neurons, the size of the respective layer does not change.

[0077] If the threshold value is lower than the current number of neurons in the respective layer, then the last neurons, corresponding to the smallest values of fai), are removed from the layer. For instance, removal of one or more neurons from a particular layer can be accomplished by changing the output dimension of

[0078]

[0079] and the input dimension of

[0080]

[0081] appropriately, until the number of neurons matches the threshold.

[0082] On the other hand, if the threshold value is greater than the current number of neurons in the respective layer, new neurons are added to the layer. For instance, this may be accomplished by adding new dimensions to the weight matrices and initialising the new weights.

[0083] Therefore, in the above example, the sizes of the AWN network’s layers can grow and shrink according to the signal received by the gradient, through the adaptation of the parametric function fai). The entire process is fully differentiable.

[0084] In an example, the resulting AWN may be truncated after training, e.g. to adapt the AWN to precise hardware resources. In this regard, once an AWN has been trained and the “right” number of neurons for the task has been identified, one can easily attain a compromise between performance and computational resources by exploiting the soft-ordering imposed on the neurons’ importance. To compress a layer ■£ it is sufficient to remove the least n important neurons, which can be obtained by removing n output dimensions from the weight matrix

[0085]

[0086] and n input dimensions from Wf. This has practically zero cost, contrarily to expensive fine-tuning and retraining steps required by other pruning / distillation techniques. In theCNN case, one may simply drop the least important channels in the convolutional layer ■£.

[0087] So far, the description has focused on MLPs, i.e. tabular data. However, according to various examples, the proposed concept can be extended to other data modalities. It is relatively straightforward to extend the proposed concept to the processing of other neural network architectures that are designed to handle various data modalities.

[0088] The first observation in this regard is that CNNs, DGNs, RNNs and Transformers make use of MLPs like the one discussed above as part of their architecture, hence it is possible to adapt the number of neurons of these MLPs during training. In the specific case of CNNs, one can also introduce a function g^(i) , with the properties described above and in connection with Fig. 3. This function gp / i) may compute the importance of the CNNs channels at layer in other words, it is possible to learn the number of channels in the convolutional layers of the CNN, by applying a broadcasted element-wise multiplication between the scalar gp^(i) and the output tensor of the i-th channel.

[0089] In other cases, such as DGNs, where a single graph convolutional layer is composed by an MLP of its own, one can i) learn the number of neurons of the MLP layers inside the graph convolutional layer and ii) adapt the size of the output and input layers of the two consecutive MLPs being employed by the respective, consecutive, graph convolutional layers. The reason for ii) is that the output of the former is the input of the latter; in this case, the output activations of the previous graph convolutional layers need to be appropriately multiplied by another monotonically decreasing function as the one described in connection with Fig.3. To put it another way, the two MLPs are virtually merged into one.

[0090] In summary, at least one of the examples of the present disclosure has at least one of the following features and / or at least one of the following advantages:

[0091] - Potentially reducing the number of hyper-parameter configurations to try by learning the number of neurons for each layer of a neural network;- Imposing a soft-ordering on the importance of neurons at each layer. This allows to instantaneously reduce the size of a layer after training by simply removing the least important neurons, achieving a trade-off between computational requirements and performances at zero extra cost.

[0092] - Easy-to-us, as the proposed concept relies on simple backpropagation; - Applicable to all kinds of architectures: MLPs, RNNs, CNNs, and DGNs; - Providing a strategic advantage in the LLMs competition, by training LLMs that are as “small” as possible without losing performances and by providing an instantaneous way to build low-power LLMs from the original one at the cost of reduced performance.

[0093] When certain aspects are mentioned in relation to a device or system, they should also be considered as descriptions of the corresponding methods. For example, a block, component, or functional aspect of the device or system may correspond to a method step or feature of the related method. Therefore, aspects described regarding a method should also be understood as depicting a corresponding element, property, or functional feature of the corresponding device or system. In simpler terms, if something is described in relation to a device or system, it can also be applied to the corresponding method, and vice versa.

[0094] Many modifications and other embodiments of the invention set forth herein will come to mind to the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.L i s t o f r e f e r e n c e s i g n s

[0095] training dataset

[0096] trained neural network

[0097] trained neural network

[0098] system

[0099] interface

[0100] processor

[0101] storage device

[0102] training process

[0103] width adaptation process

[0104] running inference with neuron’s importance providing training data

[0105] choosing neural network, functions

[0106]

[0107] and threshold

[0108] training neural network

[0109] selecting width

[0110] performing inference

[0111] computing current value of loss function computing gradients of loss function and updating parameters

[0112] outputting neural network model

[0113] truncating neural network

Claims

C l a i m s1. A computer-implemented method for developing a neural network model, the method comprising:setting up a neural network including a given number of layers ■£, wherein, during training, each neuron’s activation at layer ■£ is multiplied by a pre-defined function fa^i) for each layer where i is the position of the neuron and a{is a learnable parameter for layerusing a training dataset for training the neural network, wherein training includes at each iteration:adaptively selecting the width of the layers I of the neural network based on a pre-defined threshold value of functioncomputing the current value of the neural network’s loss function to be minimized, andcomputing gradients of the neural network’s loss function and updating the parameters of the neural network including parameterand outputting, after training, the trained neural network model for making predictions on unseen data.

2. The method according to claim 1, wherein the function fa^i) is constructed in such a way that it imposes a soft-ordering of an importance of the neurons at each layer I.

3. The method according to claim 1 or 2, wherein the function fa^i) is a monotonically decreasing function.

4. The method according to any of claim 1 to 3, wherein the functionis a distribution, the method further comprising selecting the width of a layer I of the neural network as the quantile function of the distribution evaluated at a chosen value q.

5. The method according to any of claim 1 to 4, wherein adaptively selecting the width of a layer I of the neural network based on a pre-defined threshold value of function fa^i) comprises:determining that the current number of neurons in the layer I is higher than the threshold value; andremoving the neurons corresponding to the smallest values of fa#(i) from the layer I until the number of neurons matches the threshold value.

6. The method according to any of claim 1 to 4, wherein adaptively selecting the width of the layers I of the neural network based on a pre-defined threshold value of function ^(i) comprises:determining that the current number of neurons in the layer I is lower than the threshold value; andadding neurons to the layer I until the number of neurons matches the threshold value.

7. The method according to claim 5 or 6, wherein removing and / or adding one or more neurons from / to the layer I is performed by changing the output dimension of weighting matrixand the input dimension of weighting matrixappropriately.

8. The method according to any of claim 1 to 7, further comprising:adapting the neural network model to specific hardware requirements by means of a post training truncation of the neural network model, wherein the truncation includes removing, after training, the last rows and / or columns of the weight matrices corresponding to the least important neurons.

9. The method according to any of claim 1 to 8, wherein the neural network is a Large Language Model, LLM, and wherein the training dataset includes training samples to train the LLM.

10. The method according to claim 9, wherein the training step of adaptively selecting the width of the layers I of the neural network based on a pre-definedthreshold value of functionis applied to the Multi-Layer Perceptrons, MLPs, inside transformer layers of the LLM.

11. The method according to any of claim 1 to 8, wherein the neural network is a machine-learning interatomic potentials, MLIP, architecture, and wherein the training dataset includes training samples related to a molecular system to simulate using the MLIP architecture.

12. The method according to claim 11, wherein the training step of adaptively selecting the width of the layers I of the neural network based on a pre-defined threshold value of function fa^i) is applied to the MLIP architecture.

13. The method according to any of claims 1 to 12, wherein the training step of adaptively selecting the width of the layers I of the neural network based on a predefined threshold value of function fa^i) is applied to Multi-Layer Perceptrons, MLPs, contained in a Convolutional Neural Network, CNN, a Recurrent Neural Network, RNN, or a Deep Graph Network, DGN.

14. A system (10) comprising one or more processors (14) and one or more storage devices (16), wherein the system (10) is configured to carry out the method of one of the claims 1 to 13.

15. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of one of the claims 1 to 13.

Images