Method and apparatus for training model, controller, refrigeration device, and product

Abstract

Embodiments of the present disclosure relate to a method and an apparatus for training a model, a controller, a refrigeration device, and a computer program product. The method includes determining, based on a first feature and a second feature, a predicted label by using a task processing model. The method further includes generating a reconstructed image based on the first feature. The method further includes training a first encoder, a first decoder, and the task processing model based on the predicted label, the reconstructed image, a first sample, and a sample label. According to the method in the embodiments of the present disclosure, the first encoder can be supervised by using the reconstructed image in a training process, to prevent the first encoder from losing excessive image information in a process of dimension alignment, so that the task processing model can complete a prediction task based on image information related to the prediction task, thereby improving prediction accuracy.

Description

[0001] METHOD AND APPARATUS FOR TRAINING MODEL, CONTROLLER, REFRIGERATION DEVICE, AND PRODUCT

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the field of machine learning, and more specifically, to a method and an apparatus for training a model, a controller, a refrigeration device, and a computer program product.

[0004] BACKGROUND

[0005] A machine learning technology has made significant progress in recent years. The machine learning technology constructs an algorithm model, so that a computer can learn from data and perform prediction or decision. Models based on neural network such as a convolutional neural network and a recurrent neural network have achieved breakthrough results in a plurality of fields such as image recognition and natural language processing. The development of the technology makes tasks such as image analysis, recognition, and generation more efficient and accurate, and the technology plays an important role in fields such as medical imaging diagnosis, autonomous driving, and security monitoring.

[0006] For example, a machine learning model may be used to analyze data sensed by various sensors in a refrigeration device, to predict a status of an object (for example, food) in the refrigeration device. In this way, people do not need to turn on the refrigeration device from time to time and guess the status of the object in the refrigeration device according to a subjective feeling. Models facilitate people's life.

[0007] SUMMARY

[0008] Embodiments of the present disclosure provide a method and an apparatus for training a model, a controller, a refrigeration device, and a computer program product. In a first aspect of the present disclosure, a method for training a model is provided. The method includes determining, based on a first feature and a second feature, a predicted label by using a task processing model, where the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The method further includes generating, by a first decoder, a reconstructed image based on the first feature. The method further includes training the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label.

[0009] In a second aspect of the present disclosure, a method for processing data is provided. The method includes obtaining first data and second data, where the first data belongs to an image modality, and the second data belongs to a non-image modality. The method further includes determining, based on the first data and the second data, a target prediction value by using the trained first encoder, the trained second encoder, and the trained task processing model according to the first aspect of the present disclosure.

[0010] In a third aspect of the present disclosure, an apparatus for training a model is provided. The apparatus includes a first prediction unit, configured to determine, based on a first feature and a second feature, a predicted label by using a task processing model, where the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The apparatus further includes a reconstruction unit, configured to generate, by a first decoder, a reconstructed image based on the first feature. The apparatus further includes a training unit, configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label.

[0011] In a fourth aspect of the present disclosure, an apparatus for processing data is provided. The apparatus includes an obtaining unit, configured to obtain first data and second data, where the first data belongs to an image modality, and the second data belongs to a non-image modality. The apparatus further includes a second prediction unit, configured to determine, based on the first data and the second data, a target prediction value by using the trained first encoder, the trained second encoder, and the trained task processing model according to the first aspect of the present disclosure.

[0012] In a fifth aspect of the present disclosure, a controller is provided. The controller includes at least one processor; and a memory, where the memory is coupled to the at least one processor, and the memory has instructions stored therein, where the instructions, when executed by the at least one processor, cause the controller to perform the method according to the first aspect or the second aspect of the present disclosure.

[0013] In a sixth aspect of the present disclosure, a refrigeration device is provided. The refrigeration device includes the controller according to the fifth aspect of the present disclosure.

[0014] In a seventh aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transitory computer-readable medium, and the computer program product includes machine-executable instructions, where the machine-executable instructions, when executed, cause a machine to perform the method according to the first aspect or the second aspect of the present disclosure.

[0015] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has machine-executable instructions stored therein, where the machine-executable instructions are executed by a processor to implement the method according to the first aspect or the second aspect of the present disclosure.

[0016] It should be understood that, the content described in the summary part is neither intended to identify key or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the embodiments of the present disclosure will be easily understood through the following descriptions.

[0017] BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing in more detail exemplary embodiments of the present disclosure with reference to accompanying drawings, the foregoing and other objects, features, and advantages of the present disclosure will become clearer. In the exemplary embodiments of the present disclosure, the same reference numerals usually represent the same component.

[0019] FIG. 1 is a schematic diagram of an exemplary environment in which a plurality of embodiments of the present disclosure may be implemented;

[0020] FIG. 2 is a flowchart of a method for training a model according to an embodiment of the present disclosure;

[0021] FIG. 3 is a diagram of a training process in a scenario of predicting freshness of food according to an embodiment of the present disclosure;

[0022] FIG. 4 is a schematic diagram of determining a training loss according to an embodiment of the present disclosure;

[0023] FIG. 5 is a schematic structural diagram of an image encoder according to an embodiment of the present disclosure;

[0024] FIG. 6 is a schematic diagram of an application scenario according to an embodiment of the present disclosure; FIG. 7 is a schematic diagram of a method for training a model according to an embodiment of the present disclosure;

[0025] FIG. 8 is a schematic diagram of a method for training a model according to another embodiment of the present disclosure;

[0026] FIG. 9A is a diagram of a process of a method for processing data according to an embodiment of the present disclosure;

[0027] FIG. 9B is a flowchart of a method for processing data according to an embodiment of the present disclosure;

[0028] FIG. 10 is a schematic diagram of an apparatus for training a model according to an embodiment of the present disclosure;

[0029] FIG. 11 is a schematic diagram of an apparatus for processing data according to an embodiment of the present disclosure; and

[0030] FIG. 12 is a schematic block diagram of an exemplary device applicable to implementing an embodiment of content of the present disclosure.

[0031] In the accompanying drawings, the same or corresponding numbers indicate the same or corresponding parts.

[0032] DETAILED DESCRIPTION

[0033] The following describes embodiments of the present disclosure in detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that, the present disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, but the embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that, the accompanying drawings and the embodiments of the present disclosure are merely intended for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

[0034] In descriptions of the embodiments of the present disclosure, the term "include" and similar terms should be understood as open inclusion, that is, "include but are not limited to". The term "based on" should be understood as "at least partially based on". The term "an embodiment" or "this embodiment" should be understood as "at least one embodiment". Terms "first", "second", and the like may represent different objects or the same object. The following may further include other explicit and implicit definitions.

[0035] In many fields, multi-modal samples need to be processed by using a machine learning model. Samples in different modalities may have different dimensions. Performing prediction by using samples in a plurality of modalities can fully and comprehensively use information that is related to a prediction task and that is from different angles, to perform accurate prediction. For example, a cross-attention module can receive two samples and synthesize features of the two samples, to obtain a fused feature related to the prediction task. Dimensions of features input to the cross-attention module usually need to be the same. Otherwise, a related matrix operation cannot be performed in the cross-attention module. In addition, some other deep learning modules need to perform prediction based on features having close dimensions.

[0036] If one modality is an image modality, and another modality is a non-image modality, for example, a value modality, a dimension difference between samples in the two modalities in a computer is usually great. For example, a sample in the image modality may be represented as a matrix 1920x 1080 (which depends on a resolution of the sample), where each element represents a pixel value. In addition, a sample in the non-image modality may be represented as a matrix 7x 1, where each element represents a sensing value. In a related technology, a method of compressing only dimensions of samples in an image modality is usually used to implement dimension alignment, but this causes loss of image information, reducing prediction accuracy.

[0037] In view of this, the present disclosure provides a method for training a model. According to the method, when a task processing model is trained, a capability of an encoder for reserving image information and a capability of a decoder for restoring image information are trained in parallel, and the model is trained based on an image loss and a prediction loss, that is, training of the task processing model is supervised by using the image loss. According to the method in the present disclosure, a first encoder can be supervised by using a reconstructed image in a training process, to prevent the first encoder from losing excessive image information in a process of dimension alignment, so that the task processing model can complete a prediction task based on image information related to the prediction task, thereby improving prediction accuracy.

[0038] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. FIG. 1 is a schematic diagram of an exemplary environment 100 in which a plurality of embodiments of the present disclosure may be implemented. As shown in FIG. 1, the exemplary environment 100 includes a first encoder 106, a second encoder 108, and a task processing model 114. In this embodiment, at least the first encoder 106, a first decoder 110, and the task processing model 114 are trained objects.

[0039] A predicted label 116 is determined by using the task processing model 114 based on a first feature el and a second feature e2. The first feature el is determined by the first encoder 106 based on a first sample 102 belonging to an image modality, and the second feature e2 is determined by the second encoder 108 based on a second sample 104 belonging to a non-image modality. The first feature el is determined by the first encoder 106, the second feature e2 is determined by the second encoder 108, and a dimension of the first feature el is close to or the same as a dimension of the second feature e2. In this embodiment, the first sample may be represented by using a first representation 102, and the second sample may be represented by using a second representation 104. The second representation 104 in FIG. 1 is a onedimensional vector. This is merely an example and does not constitute a limitation on the present disclosure.

[0040] In this embodiment, a dimension of the first representation 102 is usually much greater than a dimension of the second representation 104.

[0041] If the dimension of the first representation 102 is directly compressed into a dimension (for example, one dimension) that is close to or the same as the dimension of the exemplary second representation 104, a large amount of image information may be lost, resulting in inaccurate prediction. The first encoder 106 and the second encoder 108 may have a similar model structure.

[0042] A reconstructed image 112 is generated by the first decoder 110 based on the first feature el. Because the first sample is a sample in the image modality, in a subsequent prediction process, prediction theoretically should be performed based on the image information rather than other information unrelated to the image information. Therefore, to ensure that the first encoder 106 may mainly reserve the image information in a training process, the reconstructed image 112 may be generated based on the first feature el. In some embodiments, the reconstructed image 112 obtained after training should at least include image information that can fully reflect information related to a prediction task. The first feature el may be decoded by using the first decoder 110, to obtain the reconstructed image 112.

[0043] The first encoder 106, the first decoder 110, and the task processing model 114 are trained based on the predicted label 116, the reconstructed image 112, the first sample, and a sample label. The sample label is, for the prediction task, a true value corresponding to the first sample and the second sample. In a classification task, the sample label is a discrete value, and represents categories to which the first sample and the second sample belong. For example, in a prediction task of freshness, the sample label may be "0" or "1", respectively representing two different types (which are fresh or spoiled) of the freshness. In a regression task, the sample label is a continuous value, and represents a specific value that needs to be predicted. For example, in the prediction task of the freshness, the sample label indicates actual freshness (for example, the freshness is 80%). The present disclosure includes and is not limited to the foregoing task categories.

[0044] In this embodiment, the reconstructed image 112 functions as a supervision factor, and a loss degree of the image information may be determined by contrasting the reconstructed image 112 with the first sample, to train a capability of the first encoder 106 for an image information reservation task. The predicted label 116 and the sample label are used for determining a prediction loss, to train a prediction capability of the task processing model for the prediction task. In this training manner, it can be ensured that the first encoder 106 reserves the image information as much as possible while training the prediction capability, so that the prediction capability is trained based on the image information. Therefore, the first encoder and the task processing model obtained through training may perform prediction mainly based on the image information and have good prediction accuracy.

[0045] In an application scenario shown in FIG. 1, the method according to some embodiments of this application is implemented on a server. However, this is merely exemplary and is not restrictive. The method according to some embodiments of this application may alternatively be implemented on another body having sufficient computing resources and computing capabilities. In addition, the first encoder 106, the second encoder 108, and the task processing model 114 may be respectively located on different bodies.

[0046] The exemplary environment 100 in which the embodiments of the present disclosure can be implemented is described above with reference to FIG. 1. A flowchart of a method 200 for training a model according to an embodiment of the present disclosure is described below with reference to FIG. 2. Block 202: Determine, based on a first feature and a second feature, a predicted label by using a task processing model, where the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality.

[0047] A dimension of the first sample is usually much greater than a dimension of the second sample. The first sample and the second sample need to be subsequently fused by using a task processing model 114. Therefore, the first sample and the second sample need to be dimensionally aligned, to complete subsequent calculation. In the present disclosure, "alignment" indicates that dimensions of two or more features are changed into a close quantity of dimensions or a same quantity of dimensions. The task processing model 114 may further include a prediction module configured to perform prediction based on a fused feature. The prediction module may be any prediction module capable of completing a prediction task, for example, may be a prediction head including a plurality of fully-connected network layers. Both feature extraction and dimension alignment on the first sample and the second sample may be implemented by using the first encoder and the second encoder.

[0048] Block 204: Generate, by a first decoder, a reconstructed image based on the first feature. A first feature el is a feature obtained after extraction and dimension processing are performed on the first sample by means of encoding by using a first encoder 106. A first decoder 110 has a capability of improving a dimension. To ensure that subsequent training of the prediction capability is performed based on the image information rather than other information, it needs to be ensured that the first encoder 106 reserves the image information as much as possible. How much image information the first feature reserves can be seen herein by generating the reconstructed image by the first decoder 110.

[0049] Block 206: Train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label. In some embodiments, submodels included in the first encoder 106, a second encoder 108, and the task processing model 114 may be trained together based on the predicted label, the reconstructed image, the first sample, and the sample label. If the second encoder 108 is a trained model, only the first encoder 106, the first decoder 110, and the task processing model 114 may be trained.

[0050] In this operation, a reconstructed image 112 functions as a supervision factor, and a loss degree of the image information may be determined by contrasting the reconstructed image 112 with the first sample, to train a capability of the first encoder 106 for an image information reservation task. A predicted label 116 and the sample label are used for determining a prediction loss, to train a prediction capability of the first encoder 106 and the task processing model 114 for the prediction task. In this training manner, it can be ensured that the first encoder 106 reserves image information related to the prediction task as much as possible while perform dimension alignment, so that the prediction capability is trained based on the related image information. Therefore, the first encoder 106 and the task processing model 114 obtained through training may perform prediction mainly based on the image information and have good prediction accuracy.

[0051] According to the method in the present disclosure, a first encoder can be supervised by using the reconstructed image in a training process to prevent the first encoder from losing excessive image information in a process of dimension alignment, so that the task processing model can complete a prediction task based on image information related to the prediction task, thereby improving prediction accuracy.

[0052] In some embodiments, the first feature may be determined by the first encoder based on a first representation 102, the first representation 102 may be used for indicating the first sample belonging to the image modality, the second feature may be determined by the second encoder based on a second representation 104, the second representation 104 may be used for indicating the second sample belonging to the non-image modality, and the first representation 102 and the second representation 104 have different dimensions. For example, a dimension of the first representation 102 may be 100^ 100, and a dimension of the second representation 104 may be 1 x 10. The dimension of the first representation 102 may be much larger than the dimension of the second representation 104. When performing operations on two input vectors, some task processing models require that dimensions of the two vectors are completely the same. Therefore, in this embodiment, the first feature and the second feature after being aligned by using the first encoder and the second encoder have a same dimension. In this way, a crossattention module may complete the prediction task by calculating the first feature and the second feature. Optionally, the task processing model includes the cross-attention module, and the cross-attention module needs to process vectors having a same dimension. In some embodiments, the method further includes: obtaining the first representation 102 based on the first sample, and obtaining the second representation 104 based on the second sample. The first representation 102 may represent a distribution status (a pixel matrix) of pixels in the first sample belonging to the image modality, and the second representation 104 may represent a vector representation, a character string representation, and the like of the second sample belonging to the non-image modality. For the first representation, for example, pixels from the first sample may be cleaned, pixels having reference significance are reserved, and the pixel matrix is generated as the first representation based on relative positions of the pixels. For the second representation, for example, data from the second sample may be sorted based on a time sequence, and the vector representation is generated as the second representation based on a plurality of pieces of sorted data.

[0053] FIG. 3 is a diagram of a training process in a scenario of predicting freshness of food according to an embodiment of the present disclosure. 302: Receive an image sample (namely, a first sample) and a gas data (such as electronic nose data) sample (namely, a second sample) for predicting the freshness of the food, where modalities of the two samples are different. The gas data sample is data sensed by a gas sensor. The gas sensor is a sensor imitating a human olfactory system, and can detect, identify, and analyze various gases and volatile organic compounds. The gas sensor usually includes a gas-sensitive sensor array and a corresponding mode identification algorithm. The gas-sensitive sensor array is a core part of the gas sensor, and includes a plurality of chemical or physical sensors having different characteristics. The sensors have different response modes for different types of gases. The mode identification algorithm is used for processing data generated by the sensor array, and analyzes specific odor information through data analysis and a mode identification technology like principal component analysis, linear discriminant analysis, a support vector machine, or an artificial neural network. Then, the image sample and the gas data sample are processed into a representation that may be understood by a machine learning model. For example, the image sample may be converted into a pixel matrix 1920x 1080, and the gas data sample may be converted into a matrix 10x3.

[0054] 304: Encode the image sample into a first feature by using an image encoder, and encode the gas data sample into a second feature by using a non-image encoder. 306: Input the first feature and the second feature to a cross-attention module. The cross-attention module may fuse the first feature and the second feature, to enhance or enlarge a feature related to the freshness (for example, to cause the feature to have a high attention weight), reduce a feature unrelated to the freshness (for example, to cause the feature to have a small attention weight), integrate features related to the freshness in the first feature and the second feature into one feature, and the like. This helps a prediction module accurately predict the freshness. The fused feature is input to the prediction module for prediction, to obtain a predicted label.

[0055] 308: Generate, by using an image decoder (namely, the first decoder), a reconstructed image based on the first feature. For example, the image decoder may have a structure corresponding to that of the image encoder, for example, the structure is inverse to that of the image encoder. 310: Calculate a first reconstruction loss and a prediction loss. A loss function may be any appropriate loss function, for example, a cross-entropy loss function. The first reconstruction loss is determined based on a difference between the first sample and the reconstructed image, and the prediction loss is determined based on a difference between a sample label and a predicted label. A weighted sum of the first reconstruction loss and the prediction loss may be used as a training loss of current training. In some embodiments, weights of the first reconstruction loss and the prediction loss may be preset. 312: Train the image encoder, the image decoder, and the cross-attention module based on the training loss.

[0056] In this embodiment, the first reconstruction loss and the prediction loss are aggregated, to obtain the training loss that reflects both an image information extraction capability of the image encoder and a prediction capability of the task processing model, thereby training the image encoder and a prediction capability related to the cross-attention module. In this embodiment, a gas sensor may be replaced with an odor sensor, and a gas data sample is replaced with an odor data sample for prediction. The odor sensor is a measurement apparatus that measures odor intensity and represents the odor intensity as a number. When an odorant substance occurs in an environment, an odor molecule may come into contact with a sensing element of the sensor. The sensor may convert such a chemical signal into an identifiable electrical signal, to generate the odor data sample.

[0057] FIG. 4 is a schematic diagram of determining a training loss according to an embodiment of the present disclosure. An image sample xl is input to an image encoder 402, to obtain a first feature el. Anon-image sample x2 is input to a non-image encoder 404, to obtain a second feature e2. The image encoder 402 propagates the first feature el to both an image decoder 406 and a freshness prediction model 408, and the non-image encoder 404 propagates the second feature e2 to the freshness prediction model 408. The freshness prediction model 408 includes a cross-attention module and a prediction module. The image decoder 406 obtains a reconstructed image x' by decoding the first feature el, and the freshness prediction model 408 obtains a predicted label y' by performing prediction on the first feature el and the second feature e2. A first reconstruction loss LI may be obtained based on xl and x', and a prediction loss L2 may be obtained based on a sample label y and y'. Weighted summation is performed on LI and L2 based on a preset weight, to obtain a training loss L3. Parameters of the image encoder 402, the image decoder 406, and the freshness prediction model 408 are adjusted based on the training loss L3. In this embodiment, the image encoder, the image decoder, the crossattention module, and the prediction module all can be trained, and a trained model does not need to be prepared in advance.

[0058] FIG. 5 is a schematic structural diagram of an image encoder according to an embodiment of the present disclosure. In this embodiment, a first feature may be determined by using the image encoder including a network layer 502, a network layer 504, and a network layer 506. This embodiment includes inputting a first representation to a convolution layer 502 of the image encoder, to obtain a first convolution feature. This embodiment further includes inputting the first convolution feature to a pooling layer 504 of the image encoder, to obtain a first pooling feature. An exemplary pooling process is shown below the pooling layer 504, which helps adjust a dimension. In this example, a pooling manner of average pooling is used. This embodiment further includes inputting the first pooling feature to a full connected layer 506 of the image encoder, to obtain the first feature. By using the image encoder in this embodiment, data in an image modality can be effectively processed, a feature related to image information can be extracted from the data, and dimension adjustment can be implemented. It should be noted that, each network layer in the image encoder does not necessarily include only one layer, and may include a plurality of layers, for example, two full connected layers are shown in the full connected layer 506.

[0059] The image decoder may perform a corresponding inverse operation. In some embodiments, the first feature is input to an upsampling layer of the image decoder, to obtain a first upsampling feature; and the first upsampling feature is input to a deconvolution layer of the image decoder, to obtain the reconstructed image.

[0060] FIG. 6 is a schematic diagram of an application scenario according to an embodiment of the present disclosure. In this embodiment, a refrigerator 602 is used to represent a refrigeration device. This embodiment includes acquiring, by using a camera (not shown) in the refrigerator 602, an image of food 604 inside the refrigerator 602 as an image sample.

[0061] This embodiment further includes acquiring, by using a gas sensor or an odor sensor (not shown), gas data or odor data of an area 606 (that is, a refrigeration area of an upper part of the refrigerator) containing food 604 inside the refrigerator 602 as a non-image sample. With reference to another embodiment, a capability of a model for predicting freshness of the food 604 is trained by using the image sample and the non-image sample acquired in this way.

[0062] In some embodiments, light data about the food 604 inside the refrigerator 602 may alternatively be acquired as the non-image sample by using a light sensor. The application scenario is merely exemplary, and does not constitute a limitation on the present disclosure.

[0063] FIG. 7 is a schematic diagram of a method for training a model according to an embodiment of the present disclosure.

[0064] This embodiment mainly relates to a processing process of the cross-attention module. As shown in FIG. 7, after the first feature el is obtained through encoding by the image encoder, and the second feature e2 is obtained through encoding by the non-image encoder, the first feature el and the second feature e2 are propagated to a cross-attention module 700 (shown as 402 to 408 in FIG. 4). In the cross-attention module 700, 702 is performed: Obtain a query matrix, where a plurality of query vectors are used for determining a focus of attention. 704: Obtain a key matrix. The key matrix may be considered as a clue provided to the query matrix for matching. 706: Obtain a value matrix. An attention weight of each feature in the first feature el may be determined by using the query matrix and the key matrix. The value matrix is used for performing weighted summation based on the weights, to determine a feature representation corrected for the first feature el. The matrices may be randomly initialized in advance, and become accurate in a training process. After the matrices are obtained, 708 is performed: Perform a matrix operation on the query matrix and the first feature el, to obtain a first optimization feature. 710: Perform a matrix operation on the key matrix and the first feature el, to obtain a second optimization feature. 714: Perform an operation (like a dot product) on the first optimization feature and the second optimization feature, to obtain a weight matrix. The weight matrix indicates a weight of each element in the first feature el. 716: Perform normalization processing on the weight matrix. 712: Perform a matrix operation on the value matrix and the second feature e2, to obtain a third optimization feature. 720: Determine a fused feature based on the third optimization feature and a normalized weight matrix.

[0065] In this embodiment, the fused feature may be a result obtained after weighted summation is performed on the third optimization feature based on the weight matrix. The fused feature reflects features related to the prediction task in the first feature el and the second feature e2, that is, implements complementation of features related to prediction. In addition, features included in both the first feature el and the second feature e2 are enhanced. The fused feature has high accuracy, thereby helping improve prediction accuracy.

[0066] FIG. 8 is a schematic diagram of a method for training a model according to another embodiment of the present disclosure. In this embodiment, both an image encoder 808 and a non-image encoder 810 may be trained. In this embodiment, an image representation 804 is input to the image encoder 808 to obtain the first feature el. A non-image representation 806 is input to the non-image encoder 810 to obtain the second feature e2. A first decoder 802 generates a reconstructed image 812 based on the first feature el. The first reconstruction loss LI is determined based on the reconstructed image 812 and an image sample. A cross-attention module 814 and a prediction module 816 obtain the predicted label based on the first feature el and the second feature e2. This embodiment further includes generating, based on the second feature e2, reconstructed data 822 by using a second decoder 820. This embodiment further includes: determining the prediction loss L2 based on the predicted label and the sample label; determining the first reconstruction loss LI based on the reconstructed image 812 and the image sample; determining a second reconstruction loss L3 based on the reconstructed data 822 and a non-image sample; determining a second training loss 818 (for example, determining by means of weighted summation) based on the first reconstruction loss LI, the second reconstruction loss L3, and the prediction loss L2; and adjusting parameters of the first encoder 808, a second encoder 810, the first decoder 802, the second decoder 820, the cross-attention module 814, and the prediction module 816 based on the second training loss 818. In other words, when a total loss is calculated, a loss caused by the reconstructed data is considered, and then parameters of the image decoder and the non-image decoder are adjusted together. According to the method in this embodiment, both a capability of reserving valuable information and an encoding capability related to the prediction task of two encoders may be trained, which helps improve prediction accuracy. FIG. 9A is a diagram of a process of a method for processing data according to an embodiment of the present disclosure. The method for processing data corresponds to a model inference process. An image encoder, a non-image encoder, and a task processing model trained by using the method in the foregoing embodiments may be used in this embodiment. In this embodiment, image data and non-image data are obtained, the image data is converted into a first representation ml, and the non-image data is converted into a second representation m2. In this embodiment, a first feature nl is obtained through encoding the first representation ml by an image encoder 902, and a second feature n2 is obtained through encoding the second representation m2 by a non-image encoder 904. It can be learned according to the foregoing analysis that, dimensions of the first feature nl and the second feature n2 are aligned. For example, a dimension of nl is 10^ 10, and a dimension of n2 is 10x6. In addition, the first feature nl describes image information related to a prediction task in the first representation ml.

[0067] In this way, the first feature nl and the second feature n2 are input to a cross-attention module 908 in a task processing model 906, to obtain a fused feature. The fused feature is input to a prediction module 910 in the task processing model 906, to obtain a target prediction value 912. According to the method disclosed in this embodiment, a predicted value with high accuracy can be obtained by completing the prediction task. Because a trained encoder already has a capability of extracting a core feature (which is a feature of image information related to task processing) and performing dimension alignment, in the model inference process, inference needs to be completed only based on the first representation ml and the second representation m2, and a decoder does not need to be introduced to reconstruct an image or reconstruct data.

[0068] FIG. 9B is a flowchart of a method 900 for processing data according to an embodiment of the present disclosure. The method for processing data corresponds to a model inference process. Block 920: Obtain first data and second data, where the first data belongs to an image modality, and the second data belongs to a non-image modality. At an inference stage, a modality of data should be consistent with a modality of a sample at a training stage, to make a good prediction effect. Block 922: Determine, based on the first data and the second data, a target prediction value by using a trained first encoder, a trained second encoder, and a trained task processing model in the present disclosure.

[0069] In a scenario of detecting freshness of food, the target prediction value indicates prediction on the freshness of the food. In addition, an image of the food inside a refrigeration device may be acquired as the first data by using a camera, and gas data or odor data of an area containing the food inside the refrigeration device may be acquired as the second data by using a gas sensor or an odor sensor. In this case, the method for processing data may be a method for predicting the freshness of the food. In another example, the method in this embodiment of the present disclosure may be applied to a medical device. For example, an image sample may be a scanned image of a target tissue of an organism, and a non-image sample may be data indicating a change of a content of a potassium ion in the target tissue of the organism with time. A predicted label may indicate a health degree of the target tissue of the organism. A first encoder and a task processing model obtained through training in this scenario still have a good prediction capability. In this case, the method for processing data may be a method for predicting the health degree of the target tissue of the organism.

[0070] FIG. 10 is a schematic diagram of an apparatus 1000 for training a model according to an embodiment of the present disclosure. The apparatus 1000 shown in FIG. 10 includes a first prediction unit 1002, configured to determine, based on a first feature and a second feature, a predicted label by using a task processing model, where the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality. The apparatus 1000 further includes a reconstruction unit 1004, configured to generate, by a first decoder, a reconstructed image based on the first feature. The apparatus 1000 further includes a training unit 1006, configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label.

[0071] In some embodiments, the training unit 1006 includes a first loss determining unit, configured to determine a prediction loss based on the predicted label and the sample label, where the sample label is a true value corresponding to the first sample and the second sample. The training unit 1006 further includes a second loss determining unit, configured to determine a first reconstruction loss based on the reconstructed image and the first sample. The training unit 1006 further includes a third loss determining unit, configured to determine a first training loss based on the first reconstruction loss and the prediction loss. The training unit 1006 further includes a first adjusting unit, configured to adjust parameters of the first encoder, the first decoder, and the task processing model based on the first training loss.

[0072] In some embodiments, the third loss determining unit includes a first weight determining unit, configured to obtain a first weight for the first reconstruction loss. The third loss determining unit further includes a second weight determining unit, configured to obtain a second weight for the prediction loss. The third loss determining unit further includes a third loss determining subunit, configured to determine a weighted sum of the first reconstruction loss and the prediction loss as the first training loss based on the first weight and the second weight.

[0073] In some embodiments, the apparatus 1000 further includes at least one of the following: the first feature is determined by the first encoder based on a first representation, the first representation is used for indicating the first sample belonging to the image modality, the second feature is determined by the second encoder based on a second representation, the second representation is used for indicating the second sample belonging to the non-image modality, the first representation and the second representation have different dimensions, and the first feature and the second feature have a same dimension; and the task processing model includes a cross-attention module.

[0074] In some embodiments, the reconstruction unit includes an upsampling unit, configured to input the first feature to an upsampling layer of the first decoder, to obtain a first upsampling feature. The reconstruction unit further includes a deconvolution unit, configured to input the first upsampling feature to a deconvolution layer of the first decoder, to obtain the reconstructed image.

[0075] In some embodiments, the apparatus 1000 further includes a second reconstruction unit, configured to generate, based on the second feature, reconstructed data by using a second decoder. In addition, the training unit 1004 includes a fourth loss determining unit, configured to determine a prediction loss based on the predicted label and the sample label, where the sample label is a true value corresponding to the first sample and the second sample. The training unit 1004 includes a fifth loss determining unit, configured to determine a first reconstruction loss based on the reconstructed image and the first sample. The training unit 1004 includes a sixth loss determining unit, configured to determine a second reconstruction loss based on the reconstructed data and the second sample. The training unit 1004 includes a seventh loss determining unit, configured to determine a second training loss based on the first reconstruction loss, the second reconstruction loss, and the prediction loss. The training unit 1004 includes a second adjusting unit, configured to adjust parameters of the first encoder, the second encoder, the first decoder, the second decoder, and the task processing model based on the second training loss.

[0076] In some embodiments, the apparatus 1000 further includes a first acquisition unit, configured to acquire, by using a camera, an image of food inside a refrigeration device as the first sample. The apparatus 1000 further includes a second acquisition unit, configured to acquire, by using a gas sensor or an odor sensor, gas data or odor data of an area containing the food inside the refrigeration device as the second sample.

[0077] In some embodiments, the predicted label indicates freshness of the food. The apparatus 1000 further includes a third acquisition unit, configured to acquire, by using a light sensor, light data of the area containing the food inside the refrigeration device as the second sample.

[0078] FIG. 11 is a schematic diagram of an apparatus 1100 for processing data according to an embodiment of the present disclosure. The apparatus 1100 shown in FIG. 11 includes an obtaining unit 1102, configured to obtain first data and second data, where the first data belongs to an image modality, and the second data belongs to a non-image modality. The apparatus 1100 further includes a second prediction unit 1104, configured to determine, based on the first data and the second data, a target prediction value by using a trained first encoder, a trained second encoder, and a trained task processing model according to the present disclosure.

[0079] FIG. 12 is a schematic block diagram of a controller 1200 applicable to implementing an embodiment of the present disclosure. As shown in the figure, the controller 1200 includes a processor 1201, which can perform various appropriate actions and processing according to computer program instructions stored in a read-only memory (ROM) 1202 or computer program instructions loaded to a random access memory (RAM) 1203. Various programs and data needed by an operation of the controller 1200 may be further stored in a RAM 1203. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0080] The methods and processes described above may be performed by the processor 1201. For example, in some embodiments, the methods and processes described above may be implemented as a computer software program that is tangibly included in a machine-readable medium. In some embodiments, a part or all of computer programs may be loaded and / or installed on the controller 1200 through the ROM 1202. When the computer program is loaded to the RAM 1203 and is executed by the processor 1201, one or more actions of the methods and processes described above may be performed.

[0081] The present disclosure may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions used for performing aspects of the present disclosure are loaded.

[0082] The computer-readable storage medium may be a tangible device that may hold and store instructions used by an instruction execution device. For example, the computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination thereof. In a more specific example (a non-exhaustive list), the computer-readable storage medium includes: a random access memory (RAM), a readonly memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), a static random access memory (SRAM), or any appropriate combination thereof. The computer-readable storage medium used herein is not to be interpreted as an instantaneous signal, like a radio wave or other freely propagating electromagnetic waves, an electromagnetic wave propagating through a waveguide or other transmission media (for example, an optical pulse through an optical fiber cable) or an electrical signal transmitted through a wire. The computer-readable program instructions described herein may be downloaded to each computing / processing device from the computer-readable storage medium, or may be downloaded to an external computer or external storage device through a network like the Internet, a local area network, a wide area network, and / or a wireless network. The network may include a copper transmission cable, optical fiber transmission, wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or a network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions, for storage in the computer-readable storage medium in each computing / processing device.

[0083] The computer program instructions for executing the operations of the present disclosure may be assembly instructions, instruction set architecture instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or target code written in any combination of one or more programming languages. The programming languages include an object-oriented programming language like Smalltalk or C++, and a conventional procedural programming language like C programming language or a similar programming language. The computer-readable program instructions may be completely executed on a user computer, partially executed on the user computer, executed as an independent software package, partially executed on the user computer and partially executed on a remote computer, or completely executed on a remote computer or a server. For a case involving a remote computer, the remote computer may be connected to a computer of a user through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, through the Internet by means of an Internet service provider). In some embodiments, an electronic circuit is personalized and customized through state information of the computer-readable program instructions, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA). The electronic circuit may execute the computer-readable program instructions, thereby implementing all aspects of the present disclosure.

[0084] All of the aspects of the present disclosure are described herein with reference to the flowcharts and / or the block diagrams of the method, the apparatus (system), and the computer program product in the embodiments of the present disclosure. It should be understood that, each block in the flowcharts and / or the block diagrams and a combination of the blocks in the flowcharts and / or the block diagrams may be both implemented through the computer-readable program instructions.

[0085] The computer-readable program instructions may be provided to a general-purpose computer, a dedicated computer, or a processing unit of another programmable data processing apparatus, to produce a machine. In this way, the instructions, when executed through the computer or the processing unit of the another programmable data processing apparatus, generate an apparatus that implements the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams. The computer-readable program instructions may alternatively be stored in the computer-readable storage medium. The instructions enable the computer, the programmable data processing apparatus, and / or another device to operate in a specific manner. Therefore, the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing all aspects of the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams.

[0086] The computer-readable program instructions may alternatively be loaded to the computer, the another programmable data processing apparatus, or the another device, so that a series of operations and steps are performed on the computer, the another programmable data processing apparatus, or the another device, to generate a process of computer implementation. In this way, the instructions executed on the computer, the another programmable data processing apparatus, or the another device implement the functions / actions specified in one or more blocks in the flowcharts and / or the block diagrams.

[0087] The flowcharts and block diagrams in the accompanying drawings show a system architecture, functions, and operations that may be implemented by using the system, the method, and the computer program product according to a plurality of embodiments of the present disclosure. In this regard, each block in the flowcharts or the block diagrams may represent a module, a program segment, or a part of instructions. The module, program segment, or part of the instructions include one or more executable instructions for implementing a specified logic function. In some implementations used as substitutes, functions annotated in the blocks may alternatively occur in a sequence different from that annotated in the accompanying drawings. For example, actually two blocks shown in succession may be performed basically in parallel, and sometimes may be performed in a reverse sequence. This depends on a related function. It should also be noted that, each block in the block diagrams and / or the flowcharts and combinations of blocks in the block diagrams and / or the flowcharts may be implemented by using a dedicated hardware-based system configured to perform a specified function or operation, or may be implemented by using a combination of dedicated hardware and computer instructions.

[0088] The embodiments of the present disclosure have been described above, and the foregoing description is exemplary, non-exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and the spirit of the various embodiments described, many modifications and changes are apparent to a person of ordinary skill in the art. The terms used herein are selected to best explain principles of the embodiments, practical application, or improvements to technologies in the market, or enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method (200) for training a model, comprising: determining (202), based on a first feature and a second feature, a predicted label by using a task processing model, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality; generating (204), by a first decoder, a reconstructed image based on the first feature; and training (206) the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label.

2. The method according to claim 1, wherein the training (206) the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label comprises: determining a prediction loss based on the predicted label and the sample label, wherein the sample label is a true value corresponding to the first sample and the second sample; determining a first reconstruction loss based on the reconstructed image and the first sample; determining a first training loss based on the first reconstruction loss and the prediction loss; and adjusting parameters of the first encoder, the first decoder, and the task processing model based on the first training loss.

3. The method according to claim 2, wherein the determining a first training loss based on the first reconstruction loss and the prediction loss comprises: obtaining a first weight for the first reconstruction loss; obtaining a second weight for the prediction loss; and determining a weighted sum of the first reconstruction loss and the prediction loss as the first training loss based on the first weight and the second weight.

4. The method according to claim 1, wherein the method further comprises at least one of the following: the first feature is determined by the first encoder based on a first representation, the first representation is used for indicating the first sample belonging to the image modality, the second feature is determined by the second encoder based on a second representation, the second representation is used for indicating the second sample belonging to the non-image modality, the first representation and the second representation have different dimensions, andthe first feature and the second feature have a same dimension; and the task processing model comprises a cross-attention module.

5. The method according to claim 1, wherein the generating (204), by a first decoder, a reconstructed image based on the first feature comprises: inputting the first feature to an upsampling layer of the first decoder, to obtain a first upsampling feature; and inputting the first upsampling feature to a deconvolution layer of the first decoder, to obtain the reconstructed image.

6. The method according to claim 1, wherein the method further comprises: generating, based on the second feature, reconstructed data by using a second decoder; and the training (206) the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label comprises: determining a prediction loss based on the predicted label and the sample label, wherein the sample label is a true value corresponding to the first sample and the second sample; determining a first reconstruction loss based on the reconstructed image and the first sample; determining a second reconstruction loss based on the reconstructed data and the second sample; determining a second training loss based on the first reconstruction loss, the second reconstruction loss, and the prediction loss; and adjusting parameters of the first encoder, the second encoder, the first decoder, the second decoder, and the task processing model based on the second training loss.

7. The method according to claim 1, further comprising: acquiring, by using a camera, an image of food inside a refrigeration device as the first sample; and acquiring, by using a gas sensor or an odor sensor, gas data or odor data of an area containing the food inside the refrigeration device as the second sample.

8. The method according to claim 7, wherein the predicted label indicates freshness of the food.

9. A method (900) for processing data, comprising: obtaining (920) first data and second data, wherein the first data belongs to an image modality, and the second data belongs to a non-image modality; and determining (922), based on the first data and the second data, a target prediction valueby using the trained first encoder, the trained second encoder, and the trained task processing model according to any one of claims 1 to 8.

10. The method according to claim 9, wherein the target prediction value indicates prediction on freshness of food, and the obtaining first data and second data comprises: acquiring, by using a camera, an image of the food inside a refrigeration device as the first data; and acquiring, by using a gas sensor or an odor sensor, gas data or odor data of an area containing the food inside the refrigeration device as the second data, or acquiring, by using a light sensor, light data of the area containing the food inside the refrigeration device, as the second data.

11. An apparatus (1000) for training a model, comprising: a first prediction unit (1002), configured to determine, based on a first feature and a second feature, a predicted label by using a task processing model, wherein the first feature is determined by a first encoder based on a first sample belonging to an image modality, and the second feature is determined by a second encoder based on a second sample belonging to a non-image modality; a reconstruction unit (1004), configured to generate, by a first decoder, a reconstructed image based on the first feature; and a training unit (1006), configured to train the first encoder, the first decoder, and the task processing model based on the predicted label, the reconstructed image, the first sample, and a sample label.

12. An apparatus (1100) for processing data, comprising: an obtaining unit (1102), configured to obtain first data and second data, wherein the first data belongs to an image modality, and the second data belongs to a non-image modality; and a second prediction unit (1104), configured to determine a target prediction value based on the first data and the second data by using the trained first encoder, the trained second encoder, and the trained task processing model according to any one of claims 1 to 8.

13. A controller, comprising: at least one processor; and a memory, wherein the memory is coupled to the at least one processor, and the memory has instructions stored therein, wherein the instructions, when executed by the at least one processor, cause the controller to perform the method according to any one of claims 1 to 10.

14. A refrigeration device, comprising the controller according to claim 13.

15. A computer program product, wherein the computer program product is tangiblystored in a non-transitory computer-readable medium, and the computer program product comprises machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform the method according to any one of claims 1 to 10.

Images