Self-developing neural network design method inspired by DNA damage repair mechanism, and storage medium

Abstract

A self-developing neural network design method inspired by a DNA damage repair mechanism, and a storage medium. The method comprises: selecting a neural network model and defining hyperparameters thereof; acquiring an image dataset, and dividing the image dataset into a training set, a validation set and a test set, in order to form sequential tasks; on the basis of data of the first task in the training set, training the neural network model; on the basis of data of the first task in the validation set, testing a trained neural network model; performing self-developing growth on the neural network model in the width and depth directions, in order to increase the scale of the neural network model; repeating the above training and testing processes, determining whether it is necessary to further increase the scale of the neural network model, and if the increase is stopped, acquiring the classification accuracy of the model based on task data in the test set; and repeating the above process until the testing of all the task data in the test set is completed. The scale of a neural network model is dynamically adjusted during a training process, and new neurons are grown to adapt to changing tasks and environments, thereby promoting the intelligent development of neural networks in image classification tasks and the practical application thereof.

Description

A self-developing neural network design method and storage medium inspired by DNA damage repair mechanisms Technical Field

[0001] This invention relates to the field of neural network image processing technology, and in particular to a design method and storage medium for a self-developing neural network inspired by DNA damage repair mechanisms. Background Technology

[0002] In traditional neural network image classification models, the model structure is determined from the outset and cannot be changed during training and testing, which limits its practical application. In reality, data and tasks are often presented sequentially over time. Simple image classification tasks require only small-scale model structures, while complex tasks require larger-scale models. To improve model utilization and maximize performance, different network sizes should be used for different image classification tasks.

[0003] Secondly, when processing sequential tasks, neural network image classification models not only exhibit forgetting of old tasks but also a lack of learning ability, meaning their plasticity decreases. The same model achieves excellent performance on non-sequential tasks, but its overall performance drops significantly across all data when handling sequential tasks, limiting the adaptability of neural network image classification models to new tasks and environments. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a self-developing neural network design method and storage medium inspired by DNA damage repair mechanisms. This method involves dynamically adjusting the scale of the neural network model during training and growing new neurons to cope with constantly changing tasks and environments, thereby promoting the intelligent development of neural networks in image classification tasks and their practical applications.

[0005] Technical Solution: To achieve the above objectives, the present invention provides a self-developing neural network design method inspired by DNA damage repair mechanisms, comprising the following steps:

[0006] S1: Initialize the parameters and weights θ of the neural network model, and define the maximum number of times the neural network model can grow N, the performance improvement threshold ε, the model learning rate α, the number of training epochs, and the number of learning categories class_task for each task;

[0007] S2: Obtain an image dataset containing image classification labels, and based on the class_task value, divide the image dataset into a training set, a validation set, and a test set to form a sequence task;

[0008] S3: Train the neural network model based on the first task data in the training set and the learning rate α and the number of training epochs defined in S1;

[0009] S4: Based on the first task data in the validation set, test the neural network model trained in S3 and obtain the classification accuracy C of the model on the task data.

[0010] S5: Perform self-growth on the neural network model defined in S1 in both width and depth directions to increase the size of the neural network model;

[0011] S6: Repeat S3 and S4 to obtain the new model's classification accuracy C' on the task data. If C'-C>ε, repeat S3-S5 to continue increasing the size of the neural network model. If C'-C<ε and the number of times the neural network model grows in the depth direction is greater than the defined maximum number of growth times N, stop the growth of the neural network model on the current task and test it based on the first task data of the test set to obtain the model's classification accuracy T1 on the task data.

[0012] S7: Based on the second task data in the training set, repeat S3-S6 to obtain the classification accuracy T2 of the model on that task data;

[0013] S8: Repeat S7 until all image classification tasks have been tested, obtain all classification accuracies of the self-growth neural network model on the test set, and evaluate the performance of the self-growth neural network model based on the classification accuracy.

[0014] The neural network model described in S1 has an architecture of either a convolutional neural network (CNN) or a linear layer (Linear).

[0015] In S2, the sequence task partitioning is determined based on the total number of classes in the image dataset (class_num) and the number of classes learned for each task (class_task). Specifically, the training set, validation set, and test set are each divided into class_num / class_task tasks, with the number of classes for each task increasing sequentially. That is, the first task has class_task classes, the second task has 2*class_task classes, and the last task has class_num classes.

[0016] In S5, the neural network model undergoes self-growth, prioritizing growth in the width direction and then in the depth direction. Growth in the width direction involves adding new neurons or new convolutional kernel channels to the penultimate layer of the neural network model, while growth in the depth direction involves inserting new linear or convolutional layers into the penultimate layer of the neural network model.

[0017] For CNN architecture neural network models, width growth refers to increasing the number of channels in the convolutional kernel, while depth growth refers to increasing the number of convolutional layers in the entire neural network model. The number of channels in the newly added convolutional layer is set to half the number of channels in the convolutional kernel of the previous layer of the model. The last layer of the CNN architecture neural network model is the classification head, which has a fixed size and only learns weight values.

[0018] For a neural network model with a Linear architecture, width growth refers to increasing the number of neurons in the same layer, while depth growth refers to increasing the number of Linear layers in the entire neural network model. The number of neurons in the newly added Linear layer is set to half the number of neurons in the previous layer of the model.

[0019] The self-growth process of the neural network model consists of two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure with the newly added network structure. For a CNN architecture neural network model, the parameter before growth is μ. 11 =[out ch ,in ch kernel size kernel size First, increase the number of new convolutional channels: μ 12 =[new ch ,in ch kernel size kernel size ], and then concatenate them to obtain the new weight μ1 = [out ch +new ch ,in ch kernel size kernel size ];

[0020] For the Linear architecture, the parameter before growth is μ. 21 =[out fea ,in fea First, increase the number of new neurons: μ 22 =[new fea ,in fea ,], then concatenate to obtain the new weight μ2 = [out fea +new fea ,in fea ].

[0021] The present invention provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the self-developing neural network design method inspired by the DNA damage repair mechanism described above.

[0022] Beneficial effects: The present invention has the following advantages: 1. The present invention is inspired by the DNA damage repair mechanism. By dynamically adjusting the scale of the neural network model during the training process, it achieves the effect of adaptively changing the model scale according to the task complexity. At the same time, by growing a new number of neurons to cope with the constantly changing tasks and environments, it promotes the accumulation and inheritance of knowledge by the neural network model, thereby improving the robustness and adaptability of the model in the image classification process, further reducing training costs and resource consumption, and promoting the intelligent development of neural networks in image classification tasks and their practical application.

[0023] 2. This invention utilizes smaller model parameters to generate complex neural networks, alleviating the lack of plasticity in deep learning models for sequence tasks and improving the model's adaptability and learning ability for new tasks. Attached Figure Description

[0024] Figure 1 is a schematic diagram of the process of the present invention;

[0025] Figure 2 is a flowchart of the growth process of a self-developing neural network;

[0026] Figure 3 shows the training results of self-growth based on CNN architecture on the CIFAR100 dataset;

[0027] Figure 4 shows the training results of self-growth based on CNN architecture on the CIFAR10 dataset;

[0028] Figure 5 shows the feature similarity matrix between different layers of the network grown based on the Linear architecture in the CIFAR10 dataset. Detailed Implementation

[0029] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0030] As shown in Figure 1, the self-developing neural network design method inspired by DNA damage repair mechanisms of the present invention includes the following:

[0031] S1: Select either CNN or Linear neural network architecture, initialize the model's parameter weights θ, define the maximum number of network growth iterations N, the performance improvement threshold ε, the model learning rate α, the number of training epochs, the number of learning classes per task class_task, and initialize the number of depth growth iterations num_depth to 0.

[0032] S2: Based on the `class_task` value in S1, the dataset is divided into a training set, a validation set, and a test set, forming a sequence of tasks. This is determined by the total number of classes in the dataset (`class_num`) and the number of classes learned for each task (`class_task`). Specifically, the training set, validation set, and test set are each divided into `class_num / class_task` tasks, with the number of classes in each task increasing sequentially. That is, the first task has `class_task` classes, the second task has 2 * `class_task` classes, and the last task has `class_num` classes.

[0033] S3: Based on the first task data in the training set in S2, the network model defined in S1 is trained using the learning rate α and the number of training epochs in S1. The network is trained epoch times for each growth.

[0034] S4: The model trained using S3 is tested on the first task of the validation set in S2, and its classification accuracy is C.

[0035] S5: Perform self-growth growth on the network defined in S1 in both width and depth directions to increase the model size. The specific growth method is as follows: prioritize growth in the width direction, and then grow in the depth direction. For CNN architecture, width growth means increasing the number of channels in the convolutional kernel, with the number of channels increased by 5. Depth growth means increasing the number of convolutional layers in the network, with the number of channels in the new convolutional layer being half the number of channels in the previous layer. For Linear architecture, width growth means increasing the number of neurons in the same layer, with the number of neurons increased by 50. Depth growth means increasing the number of Linear layers in the network, with the number of neurons in the new Linear layer being half the number of neurons in the previous layer.

[0036] Growing in the width direction involves adding new neurons or new convolutional kernel channels to the penultimate layer of the model. Growing in the depth direction involves inserting new linear or convolutional layers into the penultimate layer of the model. The last layer of the model is the classification head, which has a fixed size and only learns its weight values.

[0037] The self-growth process of a neural network model consists of two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure with the newly added network structure. For a CNN architecture, the parameter before growth is μ1 = [out]. ch ,in ch kernel size kernel size First, increase the number of convolution channels, μ2 = [new ch ,in ch kernelsize kernel size Then concatenate them to obtain the new weight μ = [out] ch +new ch ,in ch kernel size kernel size For the Linear architecture, the pre-growth parameter is μ1 = [out] fea ,in fea First, the number of new neurons increases, μ2 = [new] fea ,in fea Then concatenate the ,] to obtain the new weight μ = [out] fea +new fea ,in fea ].

[0038] S6: Repeat steps S3 and S4 to obtain a new classification accuracy of C' for this task; if C'-C>ε, repeat steps S3-S5 to continue increasing the network size; if C'-C<ε and the number of growth steps in the depth direction num_depth is incremented by one, and if num_depth is greater than N defined in S1, stop the model growth on the current task and test it on the first task of the test set in S2 to obtain a classification accuracy of T1; as shown in Figure 2, it is a schematic diagram of the growth process of the self-developing neural network.

[0039] S7: Based on the second task data in the training set in S2, repeat steps S3-S6 to obtain its classification accuracy as T2.

[0040] S8: Repeat step S7 until all tasks have been tested and the classification accuracy T1, T2...TN of the test set is obtained.

[0041] To evaluate the rationality of the final grown network structure, after the model has learned all the data, similarity calculations are performed on the features learned between different layers of the model, and on the features between different neurons within the same layer. The calculation method is as follows:

[0042] Where, x i ,x j H represents the features of different layers of a network or the features of different neurons in the same layer of a network. H is a normalized matrix, and its calculation method is as follows:

[0043] I n Let x be an identity matrix of dimension n. iThe specific calculation method is as follows: based on the model θ after training all tasks, 1000 samples are randomly selected from the test set, and forward propagation is performed to obtain the features of different network layers. For the Linear layer, the feature dimension is (batch, fea_dim), where batch is the batch size and fea_dim is the feature, which is directly calculated using the CKA function. For the CNN layer, the obtained feature dimension is (batch, out_ch, height, width), which needs to be averaged to obtain a two-dimensional dimension, that is, the average is taken on the weight and width dimensions to obtain the (batch, out_ch) features, and then the similarity is calculated using the CKA function.

[0044] Taking the CIFAR100 and CIFAR10 datasets as examples, the self-growth training results of the CNN architecture based on the CIFAR100 dataset are shown in Figure 3, and the self-growth training results of the CNN architecture based on the CIFAR100 dataset are shown in Figure 4.

[0045] The CIFAR10 dataset contains 10 categories, with 2 categories per task, for a total of 5 tasks. The CIFAR100 dataset contains 100 categories, with 5 categories per task, for a total of 20 tasks. The seq S-DNN dataset represents the test results of the self-developing neural network on sequence tasks, the seq baseline dataset represents the test results of the traditional neural network on sequence tasks, the baseline S-DNN dataset represents the test results of the self-developing neural network on non-sequence tasks, and the baseline dataset represents the test results of the traditional neural network on non-sequence tasks. Experimental results show that the self-developing neural network alleviates the problem of lack of plasticity in neural networks when facing sequence tasks. That is, it is not that the model's expressive ability is insufficient, but rather that its learning ability declines when learning sequence tasks. Both datasets show a decline in test performance when learning sequence tasks, but the decline in performance of the self-developing neural network is slower, mitigating the drawbacks of lack of plasticity.

[0046] Figure 5 shows the self-developing network grown from the CIFAR10 dataset under the Linear architecture. The feature similarity matrix between different layers was calculated based on CKA. The results show that the similarity of the learned features between different layers is very low, which verifies the rationality of the self-developing neural network.

Claims

1. A method for designing self-developing neural networks inspired by DNA damage repair mechanisms, characterized in that, Includes the following steps: S1: Initialize the parameters and weights θ of the neural network model, and define the maximum number of times the neural network model can grow N, the performance improvement threshold ε, the model learning rate α, the number of training epochs, and the number of learning categories class_task for each task; S2: Obtain an image dataset containing image classification labels, and based on the class_task value, divide the image dataset into a training set, a validation set, and a test set to form a sequence task; S3: Train the neural network model based on the first task data in the training set and the learning rate α and the number of training epochs defined in S1; S4: Based on the first task data in the validation set, test the neural network model trained in S3 and obtain the classification accuracy C of the model on the task data. S5: Perform self-growth on the neural network model defined in S1 in both width and depth directions to increase the size of the neural network model; S6: Repeat S3 and S4 to obtain the new model's classification accuracy C' on the task data. If C'-C>ε, repeat S3-S5 to continue increasing the size of the neural network model. If C'-C<ε and the number of times the neural network model grows in the depth direction is greater than the defined maximum number of growth times N, stop the growth of the neural network model on the current task and test it based on the first task data of the test set to obtain the model's classification accuracy T1 on the task data. S7: Based on the second task data in the training set, repeat S3-S6 to obtain the classification accuracy T2 of the model on that task data; S8: Repeat S7 until all image classification tasks have been tested, obtain all classification accuracies of the self-growth neural network model on the test set, and evaluate the performance of the self-growth neural network model based on the classification accuracy.

2. The self-developmental neural network design method inspired by the DNA damage repair mechanism according to claim 1, characterized in that, The neural network model described in S1 has an architecture of either a convolutional neural network (CNN) or a linear layer (Linear).

3. The self-developmental neural network design method inspired by the DNA damage repair mechanism according to claim 1, characterized in that, The sequence task partitioning in S2 is determined based on the total number of classes in the image dataset (class_num) and the number of classes learned for each task (class_task). Specifically, the training set, validation set, and test set are each divided into class_num / class_task tasks, with the number of classes for each task increasing sequentially. That is, the first task has class_task classes, the second task has 2*class_task classes, and the last task has class_num classes.

4. The self-developmental neural network design method inspired by the DNA damage repair mechanism according to claim 1, characterized in that, In S5, the neural network model undergoes self-growth, prioritizing growth in the width direction before growing in the depth direction. Growth in the width direction involves adding new neurons or new convolutional kernel channels to the penultimate layer of the neural network model, while growth in the depth direction involves inserting new linear or convolutional layers into the penultimate layer of the neural network model.

5. The self-developmental neural network design method inspired by DNA damage repair mechanisms according to claim 4, characterized in that, For CNN architecture neural network models, width growth refers to increasing the number of channels in the convolutional kernel, while depth growth refers to increasing the number of convolutional layers in the entire neural network model. The number of channels in the newly added convolutional layer is set to half the number of channels in the convolutional kernel of the previous layer of the model. The last layer of the CNN architecture neural network model is the classification head, which has a fixed size and only learns weight values.

6. The self-developmental neural network design method inspired by the DNA damage repair mechanism according to claim 4, characterized in that, For a neural network model with a linear architecture, width growth refers to increasing the number of neurons in the same layer, while depth growth refers to increasing the number of linear layers in the entire neural network model. The number of neurons in the newly added linear layer is set to half the number of neurons in the previous layer of the model.

7. The self-developmental neural network design method inspired by the DNA damage repair mechanism according to claim 4, characterized in that, The self-growth process of a neural network model consists of two steps: the first step is to grow a new network structure, and the second step is to merge the original network structure with the newly added network structure. For a CNN architecture neural network model, the parameter before growth is μ. 11 =[out ch ,in ch kernel size kernel size First, increase the number of new convolutional channels: μ 12 =[new ch ,in ch kernel size kernel size ], and then concatenate them to obtain the new weight μ1 = [out ch +new ch ,in ch kernel size kernel size ]; For the Linear architecture, the parameter before growth is μ. 21 =[out fea ,in fea First, increase the number of new neurons: μ 22 =[new fea ,in fea ,], then concatenate to obtain the new weight μ2 = [out fea +new fea ,in fea ].

8. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods according to claims 1 to 7.

Images