Multi-task learning architecture distillation

Abstract

The present application describes a method to determine an architecture for multi-task learning based on feature similarities. The proposed method is characterized by comprising the steps of: training each model of the set of models in a single-task; perform a first evaluation of feature similarities between each single-task trained model; training each model of the set of single-task trained models in cross-tasks; perform a second evaluation of feature similarity between the set of cross-task trained models; output a multi-task learning architecture model based on feature similarities from the cross-task trained models.

Description

MULTI-TASK LEARNING ARCHITECTURE DISTILLATION

[0001] The present application describes a method to determine an architecture for multi-task learning based on feature similarities.

[0002] Neural networks that learn multiple tasks are more efficient, as part of the computations are shared, and can potentially be more accurate, as learning multiple tasks can make the shared part more robust. However, choosing how many and what components of the neural network to be shared between the tasks is not trivial. Current solutions involve searching within the space of possibilities for the combinations of shared components.

[0003] This is a costly process called neural architecture search which requires training many neural networks in order to find the best one. The two most common methods to choose which combinations to try are grid search, which divides the search space linearly, and random search, which samples from the search space uniformly (Bergstra, 2012).

[0004] Other possibilities are Bayesian search, which uses a secondary model to learn the search space and uses that secondary model to guide the sampling strategy (Bergstra, 2011), Evolutionary algorithms which apply small mutations and cross-overs between candidate models have also been used for such purposes (Liu, 2021), and Successive Halving which is able to search more widely by training each model for limited iterations and successively discarding half the models (Jamieson, 2016).

[0005] Finally, reinforcement learning is also used for neural architecture search: a secondary model is used, optimized by reinforcement learning, to guide the search sampling strategy (Jaafra, 2019).

[0006] All these strategies are costly since they involve trying many different combinations. Some works exist to reduce this search space. Gao (2020) suggests trying cross-connections between the various components of the models trained for the different tasks. These connections are penalized at the level of the optimization function so that only the stronger connections are kept.

[0007] Liu (2019) and Li (2020) use knowledge distillation to reduce the search time: an ensemble of various "teacher" models are optimized that are then used to optimize the final models that is optimized from this ensemble. Guo (2020) suggests reducing the optimizing time by building a tree with the possible combinations and having each node sample from the valid parent combination; this way the various branches are optimized within the same optimization procedure.

[0008] In any case, searching is inevitable. The herein disclosed proposal avoids this costly search procedure by taking its inspiration from the task-clustering literature (Yu, 2017). These methods attempt at understanding which models are more similar by optimizing a thin component for each one of the different tasks on top of each model and measuring how the model optimized for task i is similar to task j by how different the evaluation score is from the original model that is optimized for task j.

[0009] The present invention describes a method to determine a multi-task learning architecture model based on feature similarity analysis from a set of models, characterized by comprising the steps of: training each model of the set of models in a single-task; perform a first evaluation of feature similarities between each single-task trained model; training each model of the set of single-task trained models in cross-tasks; perform a second evaluation of feature similarity between the set of cross-task trained models; output a multi-task learning architecture model based on feature similarities from the cross-task trained models.

[0010] In a proposed embodiment of present invention, the method is characterized by each model of the set of models comprising a range of sequentially connected layers which are feed by an input of data, and provide an output based on the training of said model.

[0011] Yet in another proposed embodiment of present invention, the method is characterized by the first evaluation of feature similarities between each single-task trained model and / or the second evaluation of feature similarity between the set of cross-task trained models comprising a first deep learning model comprised of a range of sequentially connected layers, feed by an input of data, and providing a first deep learning model output; a second deep learning model comprised of a range of sequentially connected layers, feed by an input of data, and providing a second deep learning model output; a first model feature range, connected to the first deep learning model, comprising data from intermediate positions between the sequentially connected layers of said first deep learning model and the output; a second model feature range, connected to the second deep learning model, comprising data from intermediate positions between the sequentially connected layers of said second deep learning model and the output; wherein a similarity evaluation is performed between the first model feature range and the second model feature range, prior to a ranking of similarity between layers which is followed by a proposal of common layers.

[0012] Yet in another proposed embodiment of present invention, the method is characterized by the similarity evaluation comprising at least one of a structural similarity index measure or a L2-norm or a Euclidean distance or a Kullback–Leibler divergence or a cosine similarity.

[0013] Yet in another proposed embodiment of present invention, the method is characterized by the cross-task training comprising training previous trained models with different tasks and evaluate their performance.

[0014] Yet in another proposed embodiment of present invention, the method can be implemented over a computer network and / or executed by a remote administrator computer.

[0015] It is disclosed a data processing system, characterized by comprising the physical means necessary for the execution of the described method. It is also disclosed a computer program, characterized by comprising programming code and / or instructions suitable for carrying out the described method. It is also disclosed a computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

[0016] The present application describes a method to determine an architecture for multi-task learning based on feature similarities.

[0017] As mentioned on Prior art, one of the main difficulties of multi-task learning is designing a network architecture capable of responding to the multiple tasks. Typically, the proposal of the network architecture is done manually through trial and error based on the developer intuition and experience. The existing methods in the state-of-the-art for automatic architecture proposal consist in searching through a range of possibilities until the best architecture is found. This method may lead to good results, but is computationally heavy, since it requires to train several models until one achieves a good performance.

[0018] The current invention discloses a novel method to automatically propose a multi-task learning architecture given a set of deep learning networks, by taking advantage of feature extraction similarity between multiple network models for different tasks. The comparison of feature information, by their similarity, allows the proposal of an architecture with common components automatically based on the single task networks.

[0019] One of the main difficulties of multi-task learning is designing a network architecture capable of responding to multiple tasks. Typically, the proposal of the network architecture is done manually through trial and error based on the developer's intuition and experience. The existing methods in the state-of-the-art for automatic architecture proposal consist of searching through the space of possibilities until the best architecture is found. These methods may lead to good results but are computationally heavy since they require training several models until one achieves a good performance.

[0020] The advantage of the proposed similarity ranking is thus that it reduces the computationally heavy process of searching for the best multi-task architecture.

[0021] For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein.

[0022] illustrates the process to obtain an architecture proposal for multi-task learning.

[0023] illustrates the diagram of feature extraction and similarity ranking (102).

[0024] illustrates an example of an architecture proposal for multi-task learning.

[0025] With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.

[0026] The Multi-task Learning Architecture Distillation (100) is a novel method that consists of a five-step process (101, 102, 103, 104, 105), as illustrated in.

[0027] The first step (101) is to have a set of network models trained for each individual task, in which it is achieved a suitable performance. This first step is essential for this novel method to work.

[0028] The second step (102) of the Multi-task Learning Architecture Distillation (100) identifies potential feature extraction candidates from each network. This second step (102) can be analyzed more in dept in, where the output information in the intermediate layers (202.1, 203.1, 205, 402.1, 403.1, 405) of each single task network, i.e. deep learning model (200, 400), is extracted and evaluated with methods of similarity (303), such as, structural similarity index measure (SSIM), L2-norm, Euclidean distance, Kullback–Leibler divergence (KLD), cosine similarity and any other method being capable of establishing a ranking of similarity between layer outputs (304). This ranking (304) allows to identify the most equivalent layers between networks.

[0029] The third step (103) of the Multi-task Learning Architecture Distillation (100) illustrated inconsists in training the previous models from the first step (101) to perform different tasks and evaluate their performance.

[0030] A cross-task performance evaluation of the networks complements the already gathered information in the second step (102), giving an additional insight of the architecture performance for different tasks.

[0031] Note that it is not mandatory for the network to achieve good performance, since it was not designed for a different task.

[0032] The fourth step (104) illustrated inis identical to the second step (102) however the evaluation of similarity is done with the features of the network trained during the previous step, that is, the third step (103).

[0033] The information gathered in this fourth step (104) will complement the evaluation done in the second step (102) and the performance achieved for the cross-task twill indicate which network has the best capabilities to perform multi-task.

[0034] The fifth and final step (105) illustrated inaggregates the information from both second (102) and forth (104) steps and proposes an architecture based on feature similarities and the network architecture performance for cross-task. Afterwards, the multi-task architecture proposed can be trained with any conventional multi-task training.

[0035] can be considered as an example of a possible outcome of the Multi-task Learning Architecture Distillation.

[0036] For this mentioned and illustrated example, let’s consider that a first model (500) is trained with an input data (501) to perform a task A, and a second model (600) is trained with an input data (601) to perform a task B.

[0037] Now, let’s assume that, during the second step (102) of the Multi-task Learning Architecture Distillation (100), i.e., the first evaluation of feature similarity between models (102), the results show that layer A1 (502), layer A2 (503) and layer A3 (504) from the first model (500), trained to perform a task A, have a highest level of equivalence when compared with layer B1 (602) and layer B2 (603) from the second model (600) trained to perform a task B.

[0038] Again, for this example purpose, assuming that during the next cross-task evaluation step, i.e., the training of each model in a cross-task (103) and further second evaluation of feature similarity between models (104), the same similarity has been found, and that the performance from the second model (600) to perform task A was higher than the first model (500) to perform task B.

[0039] The architecture proposal using this method, like suggested by the fifth step (105), i.e., the proposed architecture model for a multi-task based on feature similarities (105) would combine all this information and propose a network architecture (700), i.e., a third model trained to perform Task A and Task B (700), using layer B1 (602) and layer B2 (603) from the second model (600) as a common component for multi-task and fork the remaining architecture to each task network, respectively.

[0040] Thus, the architecture would still place the layers that had low level of similarity for each task. In this example, the task A has a fork from the common component to the layer A4 (505) and layer A5 (506) and for the task B has a fork from the common component to the layer B3 (604) and layer B4 (605).

[0041] In the provided example, the similarity ranking of the outputs produced by the various layers found that the output processed at the end of layers A1-A2-A3 (502-503-504) and the output after layers B1-B2 (602-603) is similar (according to the similarity score that was used). For that reason, B1-B2 (602-603) is used as a common layer to produce the final architecture, which is then forked into the two pathways B1-B2-A4-A5 (602-603-505-506) and B1-B2-B3-B4 (602-603-604-605).

[0042] Another note to take into account, is the possibility to have multiple architecture proposals with this method, depending on a few parameters. For instance, from the list of similarities given in step 2 (102), first evaluation of feature similarity between models (102), and step 4 (104), second evaluation of feature similarity between models (104), from, the number of similar layers to be used in the multi-task network (700) may be defined. The illustrated architecture inshows an example of using just the most similar layers between networks. Also, although the diagrams only portray two tasks A and B, this method is conceived to design multi-task networks (700) for numerous tasks without any limitations.

[0043] The herein illustrated multi-task network (700) is just a possible embodiment of the multiple possible achievable results, and the two illustrated tasks are just for simplification purposes.

[0044] The herein proposed method to determine an architecture for multi-task learning, in one of the proposed embodiments of the invention, may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the herein described method is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc.

[0045] The proposed method can be applied whenever multiple neural networks are being deployed for multiple tasks using the same inputs and whenever reducing computational complexity is important for real-time processing or resource efficiency. Some concrete examples follow:

[0046] - Advanced driver-assistance systems (ADAS) commonly require predicting multiple tasks, such as lane detection, object recognition, and traffic sign recognition, and must be processed in real time. The proposed architecture distillation approach allows ADAS to automatically integrate and share relevant neural network layers across these tasks based on feature similarity, reducing computational load, and improving processing speed with little accuracy compromise. This results in a more efficient system capable of faster response times in dynamic driving environments, which is crucial for safety;

[0047] - Medical imaging diagnostic systems: by applying the proposed architecture, a single unified model can efficiently learn shared features across different imaging tasks, such as lesion detection, tissue classification, and organ segmentation. This multi-task setup reduces the need for separate models per imaging type;

[0048] - Natural language processing, such as customer support systems, also benefits when performing different tasks like sentiment analysis, question answering, and intent classification;

[0049] - Manufacturing quality control which requires employing visual inspection for different tasks such as defect detection, object classification, and dimensional analysis.

[0050] 100 - Multi-task learning architecture distillation process;

[0051] 101 - First step – train each model in a single task;

[0052] 102 - Second step – first evaluation of feature similarity between models;

[0053] 103 - Third step – train each model in a cross-task;

[0054] 104 - Fourth step – second evaluation of feature similarity between models;

[0055] 105 - Fifth step – propose architecture model for a multi-task based on feature similarities;

[0056] 200 - First Deep learning model;

[0057] 201 - First Deep learning model Input data;

[0058] 202 - First Deep learning model First layer;

[0059] 202.1 - First Deep learning model First layer intermediate output information;

[0060] 203 - First Deep learning model Second layer;

[0061] 203.1 - First Deep learning model Second layer intermediate output information;

[0062] 204 - First Deep learning model Third layer;

[0063] 205 - First Deep learning model output;

[0064] 301 - First model features;

[0065] 302 - Second model features;

[0066] 303 - Similarity Evaluation;

[0067] 304 - Ranking of similarity between layers;

[0068] 305 - Proposal of common layers;

[0069] 400 - Second Deep learning model;

[0070] 401 - Second Deep learning model Input data;

[0071] 402 - Second Deep learning model First layer;

[0072] 402.1 - Second Deep learning model First layer intermediate output information;

[0073] 403 - Second Deep learning model Second layer;

[0074] 403.1 - Second Deep learning model Second layer intermediate output information;

[0075] 404 - Second Deep learning model Third layer;

[0076] 405 - Second Deep learning model output;

[0077] 500 – first model trained to perform Task A;

[0078] 501 – input data of the first model trained to perform Task A;

[0079] 502 - Layer A1;

[0080] 503 - Layer A2;

[0081] 504 - Layer A3;

[0082] 505 - Layer A4;

[0083] 506 - Layer A5;

[0084] 507 – Task A Output;

[0085] 600 – second model trained to perform Task B;

[0086] 601 – input data of the second model trained to perform Task B;

[0087] 602 - Layer B1;

[0088] 603 - Layer B2;

[0089] 604 - Layer B3;

[0090] 605 - Layer B4;

[0091] 606 – Task B output;

[0092] 700 – third model trained to perform Task A and Task B;

[0093] 700.1 - Common Component;

[0094] 701 – input data of the third model trained to perform Task A and Task B;

[0095] 602 - Layer B1;

[0096] 603 - Layer B2;

[0097] 505 - Layer A4;

[0098] 506 - Layer A5;

[0099] 507 - Task A Output;

[0100] 604 - Layer B3;

[0101] 605 - Layer B4;

[0102] 606 - Task B Output.

Claims

Method to determine a multi-task learning architecture model based on feature similarity analysis from a set of models, characterized by comprising the steps of: training each model of the set of models in a single-task (101); perform a first evaluation of feature similarities between each single-task trained model (102); training each model of the set of single-task trained models in cross-tasks (103); perform a second evaluation of feature similarity between the set of cross-task trained models (104); output a multi-task learning architecture model based on feature similarities from the cross-task trained models (105).Method according to the previous claim, characterized by each model of the set of models comprising a range of sequentially connected layers which are feed by an input of data, and provide an output based on the training of said model.Method according to any of the previous claims, characterized by the first evaluation of feature similarities between each single-task trained model (102) and / or the second evaluation of feature similarity between the set of cross-task trained models (104) comprising: a first deep learning model (200) comprised of a range of sequentially connected layers, feed by an input of data (201), and providing a first deep learning model output (205); a second deep learning model (400) comprised of a range of sequentially connected layers, feed by an input of data (401), and providing a second deep learning model output (405); a first model feature range (301), connected to the first deep learning model (200), comprising data from intermediate positions between the sequentially connected layers of said first deep learning model (200) and the output (205); a second model feature range (302), connected to the second deep learning model (400), comprising data from intermediate positions between the sequentially connected layers of said second deep learning model (400) and the output (405); wherein a similarity evaluation (303) is performed between the first model feature range (301) and the second model feature range (302), prior to a ranking of similarity between layers (304) which is followed by a proposal of common layers (305).Method according to any of the previous claims, characterized by the similarity evaluation (303) comprising at least one of a structural similarity index measure or a L2-norm or a Euclidean distance or a Kullback–Leibler divergence or a cosine similarity.Method according to any of the previous claims, characterized by the cross-task training comprising training previous trained models with different tasks and evaluate their performance.Method according to any of the previous claims, characterized by being implemented over a computer network and / or executed by a remote administrator computer.Data processing system, characterized by comprising the physical means necessary for the execution of the method described in any of the previous claims.Computer program, characterized by comprising programming code and / or instructions suitable for carrying out the method described in any of the preceding claims.Computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

Images