Biological information estimation device, biological information estimation method, and recording medium

Abstract

In a biological information estimation device according to the present invention, an acquisition means acquires a biological video including a skin portion of a subject. On the basis of the biological video, an estimation means estimates a predicted value of biological information of the subject and uncertainty of the predicted value. By using the biological information estimation device to estimate biological information such as blood pressure, it is possible to support decision making relating to the lifestyle of the subject.

Description

Biological information estimation device, biological information estimation method, and recording medium

[0001] This disclosure relates to the estimation of biometric information.

[0002] Methods for estimating blood pressure from facial images have been proposed. For example, Patent Document 1 describes a method in which a subject's facial image is divided to extract multiple skin regions, a pulse wave is calculated for each skin region, and a predicted blood pressure is calculated using the pulse wave propagation time.

[0003] Japanese Patent Publication No. 2023-60172

[0004] The method described in Patent Document 1 allows for blood pressure prediction from facial images without the need for devices such as electronic blood pressure monitors. However, because users cannot know the reliability of the predicted blood pressure, it is difficult for them to determine what action to take based on the predicted blood pressure.

[0005] One of the purposes of this disclosure is to provide a risk estimation device that can estimate biological information with high accuracy based on the biological images of a subject.

[0006] In one aspect of this disclosure, the biometric information estimation device comprises: acquisition means for acquiring a biometric image including a portion of the subject's skin; and estimation means for estimating a predicted value of the subject's biometric information and the uncertainty of the predicted value based on the biometric image.

[0007] In other aspects of this disclosure, a computer-based biometric estimation method acquires a biometric image including a portion of a subject's skin, and estimates, based on the biometric image, a predicted value of the subject's biometric information and the uncertainty of said prediction.

[0008] In yet another aspect of this disclosure, the recording medium records a program that causes a computer to perform a process of acquiring biological images including parts of the subject's skin, and estimating predicted values ​​of the subject's biological information and the uncertainty of said predictions based on the biological images.

[0009] This diagram shows the overall configuration of the biometric information estimation device related to this disclosure. This is a block diagram showing the hardware configuration of the biometric information estimation device. This is a block diagram showing the functional configuration of the estimation model learning device. This diagram schematically shows the operation of the first block. This diagram schematically shows the operation of the second block. This diagram schematically shows the operation of the third block. This is a flowchart of the learning process. This is a block diagram showing the functional configuration of the biometric information estimation device. This is a flowchart of the estimation process. This is a block diagram showing the functional configuration of another biometric information estimation device. This is a flowchart of another estimation process.

[0010] Preferred embodiments of the present disclosure will be described below with reference to the drawings. In the following description, variables with a symbol superimposed on them will be indicated by adding the symbol superimposed on them. For example, a variable "X" with the symbol "∧" superimposed on it will be indicated as "X ∧ It is written as "".

[0011] <Basic Principle> Blood pressure is an important vital sign related to human health. High blood pressure increases the risk of heart, brain, and kidney diseases. Generally, mercury / electronic blood pressure monitors are used to measure blood pressure, and a cuff wrapped around the arm is compressed. However, this compression of the arm is uncomfortable for the user and inconvenient for long-term monitoring. On the other hand, methods using electrocardiograms (ECG) and photoplethysmography (PPG) are being researched as cuffless blood pressure measurement methods. These methods involve calculating pulse wave propagation time and analyzing pulse waveforms. However, ECG and PPG sensors are not widely available in ordinary households, and because they are contact-type devices, they are unsuitable for daily measurements.

[0012] Therefore, this disclosure proposes a method for estimating blood pressure from facial images using Bayesian ensemble deep learning that takes uncertainty into account. In the method proposed in this disclosure (hereinafter simply referred to as the "proposed method"), the uncertainty in blood pressure estimation from facial images is modeled using a BNN (Bayesian Neural Network). Unlike ordinary neural networks (NN), BNN can take uncertainty in deep learning into account.

[0013] Furthermore, the proposed method realizes an improvement in estimation performance through ensemble learning that estimates blood pressure using a plurality of BNNs that take as input the multimodal features extracted from face videos respectively. Specifically, the proposed method addresses the following three uncertainties. (i) Aleatoric uncertainty of data Aleatoric uncertainty of data refers to the uncertainty inherent in the data, such as noise included in sensors such as rPPG (remote PPG) and measured blood pressure values, and individual differences in the cardiovascular system. (ii) Epistemic uncertainty of the model Epistemic uncertainty of the model represents the lack of knowledge of the model. For unknown data that the model has not learned, the epistemic uncertainty of the model becomes high. (iii) Ensemble uncertainty Ensemble uncertainty represents the reliability based on multiple model predictions. When the variance of the predicted values among models is large, the ensemble uncertainty becomes high.

[0014] By addressing the above three uncertainties, the proposed method calculates the reliability of the predicted blood pressure and clarifies confident predictions and unconfident predictions. As a result, for example, the obtained reliability can be effectively utilized in actual use, such as accepting the result for a confident prediction and re-measuring with a sphygmomanometer for an unconfident prediction.

[0015] In ensemble learning, models are constructed from three different perspectives for blood pressure estimation from face videos. The first perspective is blood pressure estimation using rPPG. The proposed method extracts rPPG from multiple parts of the face and performs blood pressure estimation using both the pulse wave shape and the pulse wave propagation time characteristics.

[0016] From the second perspective, the proposed method performs blood pressure estimation using the PPG signal estimated from the face video. rPPG is extracted from the face video based on a signal processing approach, but such handcrafted features are noisy and the quality of the pulse wave signal is low. In this regard, the proposed method estimates the PPG signal (the signal obtained from a contact-type PPG sensor attached to the finger) from the face video based on deep learning and extracts the pulse wave shape precisely. The proposed method realizes high-precision blood pressure estimation by using high-quality pulse waves.

[0017] The third perspective is blood pressure estimation from facial images using facial appearance information. Conventionally, a method has been proposed that uses the age and BMI estimated from facial images for blood pressure estimation. In contrast, the proposed method directly estimates blood pressure from facial images by incorporating other appearance information (such as facial shapes related to hypertension, wrinkles, sagging, alopecia, etc.).

[0018] As described above, the proposed method estimates the final predicted blood pressure by integrating the predicted values of the three estimation models: the rPPG signal, the PPG signal estimated from facial video, and the facial image.

[0019] The biological information estimation device 100 of the proposed method can be suitably applied in the fields of medicine or healthcare. For example, the biological information estimation device 100 can be used when estimating biological information related to lifestyle diseases such as blood pressure based on a facial image taken by a user using a smartphone or the like.

[0020] <First Embodiment> [Overall Configuration] Figure 1 shows the overall configuration of the biological information estimation device according to the first embodiment. The biological information estimation device 100 estimates the biological information of a subject based on the biological image of the subject. In the following embodiments, an example of estimating the blood pressure of a subject based on the facial image of the subject will be described.

[0021] A facial image of the subject is input to the biological information estimation device 100. The biological information estimation device 100 estimates the blood pressure of the subject based on the input facial image and outputs the predicted blood pressure as the estimation result. In addition to this, the biological information estimation device 100 outputs the uncertainty of the estimation result (hereinafter simply referred to as "uncertainty"). The uncertainty includes the aforementioned data uncertainty, model uncertainty, and ensemble uncertainty.

[0022] [Hardware Configuration] Figure 2 is a block diagram showing the hardware configuration of the biometric information estimation device 100. As shown in the figure, the biometric information estimation device 100 includes a processor 11, an interface (IF) 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a database (DB) 15, and a recording medium 16. Each component is connected to the others, for example, via a bus 18.

[0023] The processor 11 is a computer such as a CPU (Central Processing Unit) and controls the entire bio-information estimation device 100 by executing a pre-prepared program. Specifically, the processor 11 can be a CPU, GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof.

[0024] Furthermore, the processor 11 loads the program stored in the ROM 13 and recording medium 16 into the RAM 14 and executes each process coded in the program. The processor 11 functions as part or all of the biometric information estimation device 100. Specifically, the processor 11 performs the learning process and estimation process described later.

[0025] IF12 transmits and receives data to and from external devices. Specifically, during the learning phase, the biometric information estimation device 100 receives learning data through IF12. The learning data consists of facial images of multiple people and their blood pressure values ​​(correct values). During the estimation phase, i.e., when estimating blood pressure, the biometric information estimation device 100 receives the subject's facial image through IF12 and outputs the blood pressure estimation result and its uncertainty to a display device or other external device.

[0026] ROM 13 stores various programs executed by the processor 11. RAM 14 is used as working memory while the processor 11 is executing various processes.

[0027] DB15 stores various algorithms, data, machine learning models, etc., that the biometric information estimation device 100 uses when it performs the learning process and estimation process described later.

[0028] The recording medium 16 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or a semiconductor memory. The recording medium 16 may be configured to be detachable from the biological information estimation device 100. The recording medium 16 stores various programs executed by the processor 11.

[0029] In addition to the above, the biometric information estimation device 100 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the operator of the biometric information estimation device 100.

[0030] [Overview of Estimation Using Bayesian Deep Learning] The proposed method introduces uncertainty in the data and model using a BNN. In a BNN, a prior distribution such as a Gaussian prior distribution W~N(0,1) is set for the weight parameters to introduce uncertainty in the NN. This replaces the NN parameters with distributions instead of using deterministic weight parameters in a general NN. The method for introducing uncertainty in the data and model using a BNN is described in the following reference 1, and its contents are incorporated into this specification as a reference.

[0031] (Reference 1) A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Proc. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.

[0032] The random output of the BNN is f WIf (x) is the model's likelihood is p(y|f W Let (x) be defined as the correct blood pressure. Since blood pressure estimation is a regression task, the proposed method uses a Gaussian distribution to calculate the likelihood p(y|f W (x)) = N(μ(x),σ(x) 2 Let ) be the model's predicted average when x is input. Also, σ(x) 2 This represents observational noise (predictive variance), which indicates the uncertainty of the data that depends on the input x. This uncertainty is called "heteroscedastic uncertainty."

[0033] Given datasets X and Y, which are the input and output of a model, Bayesian inference calculates the following posterior distribution for the weights. However, since p(Y|X) cannot be calculated analytically, the posterior distribution p(W|X,Y) is a simple distribution q* θ Approximate to (W).

[0034] The proposed method is based on variational inference using Monte Carlo (MC) dropout, and finds the q* that minimizes the Kullback-Leibler (KL) divergence with p(W|X,Y). θ (W) is calculated. In MC dropout, the model is trained by inserting dropout between each weight layer, and sampling from an approximate posterior distribution is performed by enabling dropout during testing. In addition to Monte Carlo dropout, it is also possible to use any uncertainty method such as Deep Ensembles or Laplace approximation.

[0035] The proposed method performs blood pressure estimation from facial images by performing ensemble learning using the BNN described above.

[0036] [Learning Phase] Next, the learning phase of the biometric information estimation device 100 using the BNN described above will be explained. (Learning Device) The biometric information estimation device 100 estimates blood pressure using a learned estimation model. Figure 3 is a block diagram showing the functional configuration of the estimation model learning device 200. The learning device 200 is broadly composed of a first block 210, a second block 220, and a third block 230. The first block 210 estimates blood pressure using an rPPG signal generated from a face image. The second block 220 estimates blood pressure using a PPG signal estimated from a face image. The third block 230 estimates blood pressure using a face image detected from a face image.

[0037] (1) First Block First, the first block 210 will be described. The first block 210 comprises an rPPG signal generation unit 211, a BNN 212, and an optimization unit 213. The first block 210 learns a blood pressure estimation model from the rPPG signal. Figure 4 schematically shows the operation of the rPPG signal generation unit 211 and the BNN 212 of the first block 210.

[0038] The training dataset includes face images X, systolic blood pressure (SBP) Ysbp, and diastolic blood pressure (DBP) Ydbp. Let i be the sample index and N be the number of samples.

[0039] In the first block 210, the rPPG signal generation unit 211 extracts rPPG signals from the face image, and the BNN 212 estimates blood pressure based on the rPPG signals. First, the rPPG signal generation unit 211 applies face detection processing to the face image to detect facial feature points. Then, the rPPG signal generation unit 211 extracts rPPG signals from multiple ROIs (Regions of Interest) 91 of the face. In the proposed method, the rPPG signal generation unit 211 extracts the following rPPG signals from three ROIs 91: cheek, nose, and forehead, using the signal processing method POS (Plane-Orthogonal-to-Skin). Here, "K" is the number of ROIs and "F" is the number of frames.

[0040] The signal processing technique POG is described in reference 2 below, and its contents are incorporated into this specification as a reference. (Reference 2) W. Wang, AC Den Brinker, S. Stuijk, and G. De Haan, “Algorithmic principles of remote PPG,” IEEE Transactions on Biomedical Engineering, vol. 64, no. 7, pp. 1479-1491, 2016.

[0041] By extracting rPPG signals from multiple ROIs, both pulse wave shape and pulse wave propagation time features can be utilized. Specifically, the pulse wave shape in the rPPG signal contains features related to blood pressure, while rPPG signals from ROIs at relatively far distances, such as the cheek and forehead, contain features related to pulse wave propagation time.

[0042] Next, BNN212 estimates blood pressure from the generated rPPG signal. As the backbone of BNN212, a Signal-based Network (S-Net), which is used for blood pressure estimation from rPPG signals, is employed. BNN212 outputs the predicted mean and predicted variance using S-Net. The S-Net is described in reference 3 below, and its contents are incorporated into this specification as a reference. (Reference 3) B. Wu, B. Wu, B. Tsai, and C. Hsu, “A facial-image-based blood pressure measurement system without calibration,” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1-13, 2022.

[0043] Specifically, the BNN212 outputs the following four things:

[0044]

[0045] Also, BNN212 enables dropout after each 2D residual block of S-Net during learning and testing based on MC dropout, and samples an approximate posterior distribution W ∧ ~q θ (W).

[0046] The optimization unit 213 optimizes BNN212 based on the following loss function L ｒＰＰＧ (θ).

[0047] In Equation (5), the terms corresponding to the predicted variance Vs1 of SBP and the terms corresponding to the predicted variance Vd1 of DBP (see Equation (4)) enable learning of the model considering data uncertainty. Specifically, the optimization unit 213 ignores the regression losses of the first and third terms of Equation (5) when the predicted variance is large, and emphasizes the regression losses of the first and third terms of Equation (5) when the predicted variance is small to optimize BNN212. In blood pressure estimation, there is uncertainty depending on data due to sensor noise and individual differences in the cardiovascular system, and the term of the predicted variance handles these uncertainties. Although it is difficult to define the correct label for data uncertainty, in the proposed method, by optimizing the BNN based on the loss function, it is possible to automatically define data uncertainty (predicted variance). In addition to the above loss function, any loss function considering the predicted variance can be used.

[0048] The second and fourth terms of Equation (5) prevent the model from having infinite uncertainty for all samples and the loss from becoming zero. In actual learning, to numerically stabilize the predicted variance, is used, and BNN212 is optimized based on the following loss function L ｒＰＰＧ (θ).

[0049] By the above equation, the following two values become scalar values, and the possibility of division by zero can be avoided.

[0050] Thus, the optimization unit 213 learns a blood pressure estimation model from the rPPG signal using the loss function L ｒＰＰＧ (θ).

[0051] (2) Second Block Next, the second block 220 will be described. The second block 220 comprises a PPG signal estimation unit 221, a BNN 222, and an optimization unit 223. The second block 220 generates a PPG signal from a facial image and learns a model to estimate blood pressure from the PPG signal. Figure 5 schematically shows the operation of the PPG signal estimation unit 221 and the BNN 222 of the second block 220.

[0052] The rPPG signal is a handcrafted feature extracted using a signal processing approach, and the quality of the pulse wave signal may be low. In the second block, in order to precisely extract the pulse wave shape, the PPG signal acquired from a contact-type PPG sensor is used to train the model. Specifically, the second block 220 uses the following PPG signal

[0053] The estimation performance is improved by estimating facial features from facial images and blood pressure from PPG signals.

[0054] While the PPG signal accurately reflects the pulse wave shape, the rPPG signal also includes features related to pulse wave propagation time; therefore, the PPG signal and the rPPG signal contain complementary information regarding blood pressure. First, the PPG signal estimation unit 221 extracts RGB signals from facial images and estimates the PPG signal based on deep learning, following a heart rate estimation method from facial images. A heart rate estimation method from facial images is described, for example, in the following reference 4, and its contents are incorporated into this specification as a reference.

[0055] (Reference 4) Y. Akamatsu, T. Umematsu, and H. Imaoka, “CalibrationPhys: Self-supervised video-based heart and respiratory rate measurements by calibrating between multiple cameras,” IEEE Journal of Biomedical and Health Informatics, vol. 28, pp. 1460-1471, 2024.

[0056] The PPG signal estimation unit 221 applies face detection processing to the face image and detects facial feature points. Using the detected facial feature points, the PPG signal estimation unit 221 defines 16x14 blocks for the area below the eyes and creates multiple ROIs by arranging them. Subsequently, the PPG signal estimation unit 221 creates the following spatiotemporal map using the time-series signals of the three channels (red, blue, and green) in the multiple ROIs.

[0057] Here, "3" is the number of channels, K is the number of ROIs of the face image including r1, ..., rK, and F is the number of frames.

[0058] The proposed method uses a two-dimensional representation in the spatial and temporal directions, which reduces computational cost compared to known PPG signal estimation methods that use a three-dimensional representation in the spatial and temporal directions (width × height).

[0059] Specifically, the PPG signal estimation unit 221 uses PhysNet2D as a deep learning model and generates the spatiotemporal map X shown in equation (11). rgb i From the following PPG signal It generates and outputs to BNN222. Note that D ｐ This is the length of the PPG signal.

[0060] As shown in Figure 5, the BNN222 generates the VPG signal, which is the first derivative of the PPG signal, and the APG signal, which is the second derivative of the PPG signal, from the PPG signal.

[0061] In the blood pressure estimation from the PPG signal in the second block, BNN222 uses the PPG signal, VPG signal, and APG signal to estimate blood pressure in order to capture the characteristics of the pulse wave shape in more detail.

[0062] To estimate blood pressure from pulse waves, highly accurate estimation of the pulse waveform is required. Therefore, the optimization unit 223 uses a loss function that minimizes the error in the pulse waveform based on the mean squared error (MSE). In addition to this loss function, a loss function related to the pulse wave frequency can also be used. Furthermore, BNN optimization (learning using the predicted mean and predicted variance as shown in equation (8) above) can also be performed with this loss function.

[0063] Specifically, the optimization unit 223 uses the following loss function L in order to precisely estimate the pulse wave shape. pulse Use this.

[0064] Note, p i , p' i ɪp'' i This indicates the correct answer, p ^ i , p ^ ' i , p ^ `` i The values ​​shown are the predicted values ​​from the PPG signal estimation unit 221. α, β, and γ are hyperparameters that adjust the weights of the respective losses.

[0065] As shown in Figure 5, BNN222 estimates blood pressure from PPG, VPG, and APG signals. ResNet1D, which is used for blood pressure estimation from PPG signals, is used as the backbone of BNN222. ResNet1D is described, for example, in the following reference 5, and its contents are incorporated into this specification as a reference. (Reference 5) F. Schrumpf, P. Frenzel, C. Aust, G. Osterhoff, et al., “Assessment of deep learning based blood pressure prediction from PPG and rPPG signals,” in Proc. IEEE / CVF Conf. Computer Vision and Pattern Recognition Workshop (CVPRW), 2021, pp. 3820-3830.

[0066] BNN222 concatenates the feature representations obtained from each ResNet1D that takes PPG, VPG, and APG signals as inputs, and outputs the following four:

[0067]

[0068] Furthermore, BNN222 enables dropout in ResNet1D during training and testing based on MC dropout, and approximates the posterior distribution W^~q θ Sample (W). In actual training, in order to numerically stabilize the predictive variance, Let the following loss function L ＰＰＧ The BNN222 is trained based on (θ).

[0069] According to the above formula, the following two values ​​are non-zero positive scalar values, thus avoiding the possibility of division by zero. In addition to the loss function described above, any loss function that takes predictive variance into account can be used.

[0070] Thus, the optimization unit 223 calculates the loss function L pulse +L PPG Using (θ), a blood pressure estimation model is trained from facial images via pulse wave shape. Specifically, the optimization unit 223 uses the loss function L pulse +L PPG (θ) is used to optimize the networks that constitute the PPG signal estimation unit 221 and BNN 222.

[0071] (3) Third Block Next, the third block 230 will be described. The third block 230 comprises a face detection unit 231, a BNN 232, and an optimization unit 233. The third block 230 learns a model for estimating blood pressure from face images. Figure 6 schematically shows the operation of the BNN 232 in the third block 230.

[0072] Block 3, 230, estimates blood pressure from a facial image using facial appearance information. The facial image includes information related to blood pressure, such as age, BMI (Body Mass Index), facial shape, wrinkles, sagging, and alopecia, providing useful information for blood pressure estimation. Furthermore, facial appearance information has complementary information to pulse wave information, and the performance of blood pressure estimation can be improved by ensemble learning of these two types of information.

[0073] First, the face detection unit 231 applies face detection processing to the face image and detects the following face image.

[0074] Here, H is the height of the image and W is the width of the image.

[0075] Furthermore, since hair condition is related to high blood pressure, the face detection unit 231 acquires a face image that captures the entire face and outputs it to the BNN 232.

[0076] BNN232 estimates blood pressure from facial images. Specifically, it uses ResNet50 as the backbone of BNN232. BNN232 outputs the following four things:

[0077]

[0078] Furthermore, BNN232 enables ResNet50 dropout during training and testing based on MC dropout, and samples the approximate posterior distribution.

[0079] In blood pressure estimation from facial images, pre-training of the neural network (NN) is essential due to the limited number of facial images available for training. In this embodiment, a pre-trained model for facial representation learning, adaptable to various facial tasks, is used. Specifically, the facial representation learning method uses unlabeled facial images to perform self-supervised learning based on SwAV to acquire feature representations related to facial representation. By fine-tuning the pre-trained model for the blood pressure estimation task, highly accurate blood pressure estimation from facial images can be expected. An example of a facial representation learning method is described in reference 6 below, and its contents are incorporated into this specification as a reference. Furthermore, SwAV is described in reference 7 below, and its contents are incorporated into this specification as a reference.

[0080] (Reference 6) A. Bulat, S. Cheng, J. Yang, A. Garbett, et al., “Pre-training strategies and datasets for facial representation learning,” in Proc. European Conf. Computer Vision (ECCV). Springer, 2022, pp. 107-125. (Reference 7) M. Caron, I. Misra, J. Mairal, P. Goyal, et al., “Unsupervised learning of visual features by contrasting cluster assignments,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2020, vol. 33, pp. 9912-9924.

[0081] Furthermore, during the training phase, it is possible to apply data augmentation to images, such as random cropping, horizontal flipping, color shifting, and grayscale conversion. Data augmentation increases the variety of training data, making it possible to create a model that is robust to any type of data.

[0082] The optimization unit 233, similar to the first and second blocks, Let the following loss function L img Fine-tuning of the BNN232 is performed based on (θ).

[0083] In addition to the loss function described above, any loss function that takes predictive variance into account can be used.

[0084] (Learning Process) Next, the learning process of the blood pressure estimation model by the learning device 200 will be described. Figure 7 is a flowchart of the learning process of the blood pressure estimation model. This process is realized by the processor 11 shown in Figure 2 executing a pre-prepared program and operating as each element shown in Figure 3.

[0085] First, the learning device 200 acquires facial images included in the training data (step S10). Next, in the first block 210, the rPPG signal generation unit 211 generates an rPPG signal from the facial images (step S11). Next, the BNN 212 calculates the predicted mean and predicted variance of blood pressure from the rPPG signal and outputs them to the optimization unit 213 (step S12). Next, the optimization unit 213 uses the correct blood pressure values ​​included in the training data, the predicted mean, and the predicted variance to calculate the loss function L rppg (θ) is calculated (step S13), and the loss function L rppg The BNN212 is optimized based on (θ) (step S14).

[0086] Furthermore, in the second block 220, the PPG signal estimation unit 221 estimates the PPG signal from the facial image (step S21). Next, the BNN 222 calculates the predicted mean and predicted variance of blood pressure from the PPG signal and outputs them to the optimization unit 223 (step S44). Next, the optimization unit 223 uses the correct blood pressure values ​​included in the training data, the predicted mean, and the predicted variance to calculate the loss function L pulse and L ppg (θ) is calculated (step S23). Then, the optimization unit 223 calculates the loss function L pulse +L ppg Based on (θ), the PPG signal estimation unit 221 and BNN 222 are optimized (step S24).

[0087] Furthermore, in the third block 230, the face detection unit 231 detects a face image from the face video (step S31). Next, the BNN 232 calculates the predicted mean and predicted variance of blood pressure from the face image and outputs them to the optimization unit 233 (step S32). Next, the optimization unit 233 uses the correct blood pressure values ​​included in the training data, the predicted mean, and the predicted variance to calculate the loss function L img (θ) is calculated (step S33), and the loss function L img The BNN232 is optimized based on (θ) (step S34). Then the learning process is completed.

[0088] [Estimation Phase] Next, the estimation phase of the biological information estimation device 100 will be described. In the estimation phase, the blood pressure of the subject is estimated using the estimation model learned in the learning phase described above.

[0089] (Biometric Information Estimation Device) The biometric information estimation device 100 estimates blood pressure using a trained estimation model. Figure 8 is a block diagram showing the functional configuration of the biometric information estimation device 100. The biometric information estimation device 100 is broadly composed of a first block 110, a second block 120, a third block 130, and an integration unit 140. The first block 110 estimates blood pressure using an rPPG signal generated from a facial image. The second block 120 estimates blood pressure using a PPG signal estimated from a facial image. The third block 130 estimates blood pressure using a facial image detected from a facial image. The integration unit 140 integrates the estimation results from the first block 110 to the third block 130, outputs a predicted blood pressure, and also outputs the uncertainty of that predicted blood pressure.

[0090] The first block 110 comprises an rPPG signal generation unit 111 and a BNN 112. The rPPG signal generation unit 111 has the same configuration as the rPPG signal generation unit 211 in the learning device 200 shown in Figure 3. The BNN 112 is a trained BNN obtained by optimizing the BNN 212 during the learning phase.

[0091] The second block 120 comprises a PPG signal estimation unit 121 and a BNN 122. The PPG signal estimation unit 121 is obtained by optimizing the PPG signal estimation unit 221 during the learning phase. The BNN 122 is a trained BNN obtained by optimizing the BNN 222 during the learning phase.

[0092] The third block 130 comprises a face detection unit 131 and a BNN 132. The face detection unit 131 has the same configuration as the face detection unit 231 in the learning device 200 shown in Figure 3. The BNN 132 is a trained BNN obtained by optimizing the BNN 232 during the learning phase.

[0093] The integration unit 140 integrates the predicted blood pressure values ​​output by BNN112, BNN122, and BNN132 to calculate the predicted blood pressure. The integration unit 140 also calculates the uncertainty of the predicted blood pressure based on the predicted mean and predicted variance of the blood pressure output by BNN112, BNN122, and BNN132.

[0094] First, let's explain how to calculate predicted blood pressure. For the sake of simplicity, we will only explain SBP below, but the same method can be used to estimate blood pressure values ​​for DBP.

[0095] When estimating blood pressure, the subject's facial image is input to the biometric information estimation device 100.

[0096] Specifically, in the first block 110, the rPPG signal generation unit 111 generates an rPPG signal based on the facial image, and the BNN 112 outputs the predicted average blood pressure Ms1 and predicted variance Vs1 to the integration unit 140 based on the rPPG signal. Here, the BNN 112 performs T samplings based on MC dropout and outputs the average value of these as the following predicted average Ms1.

[0097] In the second block 120, the PPG signal estimation unit 121 estimates the PPG signal based on the facial image, and the BNN 112 outputs the predicted average blood pressure Ms2 and predicted variance Vs2 to the integration unit 140 based on the PPG signal. Similar to the first block 110, the BNN 122 performs T samplings based on MC dropout and outputs the average value of these samplings as the predicted average Ms2.

[0098] In the third block 130, the face detection unit 131 detects a face image from the face video, and the BNN 132 outputs the predicted average blood pressure Ms3 and predicted variance Vs3 to the integration unit 140 based on the face image. Similar to the first block 110, the BNN 132 performs T samplings based on MC dropout and outputs the average of these as the predicted average Ms3.

[0099] The integration unit 140 calculates the final predicted blood pressure y for SBP by using the following formula: the average of the predicted averages Ms1 to Ms3 output by the first block 110 to the third block 130. ^ sbpIt outputs as follows. The integration unit 140 similarly outputs the final predicted blood pressure y for DBP. ^ dbp Calculate and output the result.

[0100]

[0101] Next, we will explain how uncertainty is calculated. First, we will explain how data uncertainty is calculated. In BNN, it is possible to define data uncertainty and model uncertainty. Data uncertainty is defined by the predictive variance used in equation (5) of the loss function. First, the integration unit 140 calculates the data uncertainty U regarding the SBP estimation from the rPPG signal based on the predictive variance Vs1 from the rPPG signal. data1 This is calculated using the following formula.

[0102] Here, the right-hand side of equation (25) represents the average of the predicted variances of SBP estimated from the PPG signal over T samplings.

[0103] Similarly, the integration unit 140 determines the data uncertainty U regarding SBP estimation from the PPG signal based on the predicted variance Vs2 from the PPG signal. data2 Calculate the uncertainty U. data2 This is expressed by changing the subscript "rppg" to "ppg" in equation (25). Furthermore, the integration unit 140 calculates the uncertainty U of the data related to SBP estimation from the face image based on the predicted variance Vs3 from the face image signal. data3 Calculate the uncertainty U. data3 This can be expressed by changing the subscript "rppg" in equation (25) to "img".

[0104] Next, we will explain how to calculate the model uncertainty. The model uncertainty is defined as the variance of the prediction mean over T samplings. The integration unit 140 calculates the model uncertainty U for SBP estimation from the rPPG signal. model1 This is calculated using the following formula.

[0105]

[0106] Similarly, the integration unit 140 calculates the uncertainty U of the model for SBP estimation from the PPG signal based on the predicted variance Vs2 from the PPG signal.model2 Calculate the uncertainty U. model2 This is expressed by changing the subscript "rppg" to "ppg" in equation (26). Furthermore, the integration unit 140 calculates the uncertainty U of the model for SBP estimation from the face image based on the predicted variance Vs3 from the face image signal. model3 Calculate the uncertainty U. model3 This can be expressed by changing the subscript "rppg" in equation (26) to "img".

[0107] Next, the calculation of ensemble uncertainty will be explained. The integration unit 140 calculates ensemble uncertainty using three models. The three models are a blood pressure estimation model from rPPG signals, a blood pressure estimation model from PPG signals estimated from face images, and a blood pressure estimation model from face images. Ensemble uncertainty is defined as the variance of the predicted mean from rPPG signals, the predicted mean from PPG signals estimated from face images, and the predicted mean from face images. Specifically, the integration unit 140 calculates the ensemble uncertainty U related to SBP estimation. ens This is calculated using the following formula.

[0108]

[0109] The integration unit 140 then uses the sum of the three types of uncertainties described above as the final uncertainty. That is, the final uncertainty U regarding SBP. sbp It is calculated using the following formula.

[0110]

[0111] Based on the above, uncertainty U sbp When the value is large, the reliability of the predicted SBP is low, and uncertainty U sbp If the value is small, the reliability of the predicted SPB can be considered high. Similarly, the final uncertainty U can also be considered for DBP. dbp This can be calculated. Note that, while the final uncertainty was calculated using data uncertainty, model uncertainty, and ensemble uncertainty above, it is also possible to calculate the final uncertainty using only some of these.

[0112] In this way, understanding the reliability of predictions through uncertainty allows for flexible handling of predicted blood pressure in practical applications. For example, if the prediction reliability is low, it becomes possible to provide users with more accurate blood pressure values ​​by prompting them to remeasure or switching to measurement using medical devices such as blood pressure monitors. Furthermore, while estimation accuracy may vary depending on skin color and race, evaluating the reliability of predictions using uncertainty as an indicator can be used to identify these performance differences.

[0113] (Estimation Process) Next, the blood pressure estimation process by the biological information estimation device 100 will be described. Figure 9 is a flowchart of the blood pressure estimation process. This process is realized when the processor 11 shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 8.

[0114] First, the biometric information estimation device 100 acquires a facial image of the subject (step S40). Next, in the first block 110, the rPPG signal generation unit 111 generates an rPPG signal from the facial image (step S41). Next, the BNN 112 calculates the predicted mean and predicted variance of blood pressure from the rPPG signal and outputs them to the integration unit 140 (step S42).

[0115] Furthermore, in the second block 120, the PPG signal estimation unit 121 estimates the PPG signal from the facial image (step S43). Next, the BNN 122 calculates the predicted mean and predicted variance of blood pressure from the PPG signal and outputs them to the integration unit 140 (step S44).

[0116] Furthermore, in the third block 130, the face detection unit 131 detects a face image from the face video (step S45). Next, the BNN 132 calculates the predicted mean and predicted variance of blood pressure from the face image and outputs them to the integration unit 140 (step S46).

[0117] Next, the integration unit 140 uses the predicted mean and predicted variance of blood pressure output from BNNs 112, 122, and 132 to calculate and output the final predicted blood pressure and uncertainty (step S47). Then the estimation process is completed.

[0118] [Effects] The proposed method is highly promising for routine blood pressure monitoring and telemedicine because it can utilize readily available webcams and smartphones and allows for non-contact, video-based measurement. Furthermore, while conventional signal processing approaches for extracting pulse waves result in low-quality pulse wave signals, the proposed method uses deep learning to extract pulse waves, enabling the acquisition of high-quality pulse waves. Additionally, by modeling uncertainty, the reliability of predicted blood pressure can be calculated, clearly distinguishing between confident and uncertain predictions. This allows for actions such as prompting re-measurement with a blood pressure monitor in the case of uncertain predictions.

[0119] [Examples of application] The proposed method can be used, for example, for monitoring patients' vital signs in online medical consultations. It can also be applied to self-care checks based on vital sign estimation from facial images in the medical field, monitoring the elderly, assessing the condition of athletes, and driver monitoring.

[0120] [Modified Examples] Modified examples of the above embodiment are described below. The following modified examples can be combined as appropriate and applied to the above embodiment.

[0121] (Variations regarding input data) In the above embodiment, facial images are used as input data, but the application of the proposed method is not limited to this. For example, images other than facial images may be used, such as images of hands, as long as they show the skin of any part of the human body. Also, multiple images may be used, such as facial images and images of hands.

[0122] In the above embodiment, the RGB signals of the facial image are used to estimate blood pressure from the PPG signal, but the application of the proposed method is not limited to this, and any one or two channels of the R, B, and G signals may be used.

[0123] In the above embodiment, the third block estimates blood pressure from a facial image, but the application of the proposed method is not limited to this. For example, any biological image, such as a hand image, may be used in addition to a facial image, and multiple such images may be used.

[0124] In addition to the first to third blocks in the above embodiment, pulse wave propagation time, heart rate, heart rate variability, age, BMI, etc. estimated from facial images may be input to the NN to estimate blood pressure, and the estimation result may be ensemble-processed with the outputs of the first to third blocks to obtain the final predicted blood pressure and uncertainty.

[0125] Furthermore, although the above embodiment only mentioned data augmentation related to facial images, it is also possible to apply arbitrary data augmentation to rPPG signals and PPG signals during training. Data augmentation increases the variety of training data, making it possible to create a model that is robust to any data.

[0126] Furthermore, the extraction of the rPPG signal in the first block in the above embodiment can also be performed by any method other than POS. For example, the rPPG signal may be extracted using a deep learning model that has been trained to generate an rPPG signal from images of multiple regions of a face.

[0127] Furthermore, multi-task learning is possible, allowing for the simultaneous output of multiple arbitrary biometric data points (e.g., blood pressure, blood glucose level, cholesterol level). In this case, the number of outputs will be the number of biometric data points × 2 (predicted mean and predicted variance). For example, if there are four biometric data points—SBP, DBP, blood glucose level, and cholesterol level—there will be eight outputs. Note that in this case, the number of terms in the loss function increases as the number of biometric data points increases, but the optimization method remains the same regardless of the number of biometric data points.

[0128] (Modifications regarding output) In the above embodiment, the biological information estimation device 100 estimates blood pressure, but it is possible to output not only blood pressure but also any other biological information (biological information related to pulse wave and facial appearance), such as blood glucose level, cholesterol level, fatty liver, and lipids.

[0129] (Variations of deep learning models) Any CNN or Transformer can be used as the deep learning model (NN) used in the above embodiment.

[0130] Furthermore, the biological information generation device 100 can not simply average the predicted blood pressure from multiple models, but can also integrate the predicted blood pressure by assigning arbitrary weights to the predicted blood pressure from each model.

[0131] Furthermore, the bio-information estimation device 100 can integrate the predicted blood pressures based on the uncertainties of the models, rather than simply averaging the predicted blood pressures from multiple models. For example, the integration unit 140 may weight the predicted blood pressures output by each model using weights corresponding to the uncertainty of the predicted blood pressure of each model to obtain the final predicted blood pressure. Specifically, any integration method can be applied such that the weight of the predicted blood pressure is increased when the uncertainty is small, and decreased when the uncertainty is large.

[0132] In the above embodiment, the integration unit 140 integrates the predicted blood pressure of the three estimation models, but instead, the feature representations generated by the three estimation models may be merged within the NN, and the blood pressure may be estimated based on the obtained feature representation.

[0133] Furthermore, the training data related to blood pressure is unbalanced, with limited training data corresponding to hypertension and hypotension. Therefore, it is possible to address the imbalance in the training data by balancing it through upsampling or downsampling, or by increasing the weight of the loss function for regions with less data. Additionally, it is possible to effectively address data imbalance by dynamically changing the balance of the training data and the weight of the loss function for regions with less data as training progresses.

[0134] <Second Embodiment> Figure 10 is a block diagram showing the functional configuration of the biological information estimation device according to the second embodiment. The biological information estimation device 70 comprises an acquisition means 71 and an estimation means 72.

[0135] Figure 11 is a flowchart of the processing performed by the biometric information estimation device of the second embodiment. The acquisition means 71 acquires a biometric image including a portion of the subject's skin (step S71). The estimation means 72 estimates the predicted value of the subject's biometric information and the uncertainty of the predicted value based on the biometric image (step S72).

[0136] According to the second embodiment of the biological information estimation device 70, biological information can be estimated with high accuracy based on the biological image of the subject.

[0137] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0138] (Note 1) A biological information estimation device comprising: acquisition means for acquiring biological images including the skin portion of a subject; and estimation means for estimating predicted values ​​of the subject's biological information and the uncertainty of said predicted values ​​based on the biological images.

[0139] (Note 2) The biological information estimation device according to Note 1, wherein the estimation means estimates pulse waves from the biological image and estimates the predicted value and the uncertainty using the estimated pulse waves.

[0140] (Note 3) The bio-information estimation device described in Note 2, wherein the estimation means detects a face image from the bio-image and estimates the predicted value and the uncertainty using the detected face image.

[0141] (Note 4) The bio-information estimation device according to Note 1, comprising: a first estimation means for calculating a first predicted mean and a first predicted variance of the bio-information using pulse waves estimated from the bio-image; a second estimation means for calculating a second predicted mean and a second predicted variance of the bio-information using a face image detected from the bio-image; a predicted value calculation means for calculating a predicted value of the bio-information using the first predicted mean and the second predicted mean; and an uncertainty calculation means for calculating the uncertainty using the first predicted variance and the second predicted variance.

[0142] (Note 5) The uncertainty calculation means is a bio-information estimation device according to Note 4 that calculates the uncertainty based on the first predictive variance and the second predictive variance.

[0143] (Note 6) The bio-information estimation device described in Note 4, wherein the uncertainty calculation means calculates the uncertainty based on the variance of the first predicted mean obtained by multiple samplings and the variance of the second predicted mean obtained by multiple samplings.

[0144] (Note 7) The bio-information estimation device according to Note 4, wherein the uncertainty calculation means calculates the uncertainty based on the variance between the first predicted mean and the second predicted mean.

[0145] (Note 8) The biological information estimation device according to Note 4, comprising: the biological image being a facial image of the subject, the biological information being blood pressure, a first optimization means for optimizing the deep learning model constituting the first estimation means based on the first predicted mean and the first predicted variance, and a second optimization means for optimizing the deep learning model constituting the second estimation means based on the second predicted mean and the second predicted variance.

[0146] (Note 9) A method for estimating biological information performed by a computer, comprising: acquiring a biological image including a portion of the subject's skin; and estimating a predicted value of the subject's biological information and the uncertainty of the predicted value based on the biological image.

[0147] (Note 10) A recording medium that records a program that causes a computer to perform a process to acquire biological images including parts of the subject's skin, and to estimate predicted values ​​of the subject's biological information and the uncertainty of said predicted values ​​based on the biological images.

[0148] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference.

[0149] This application claims priority based on Japanese Patent Application No. 2024-195218, filed on 7 November 2024, and incorporates all of its disclosures herein.

[0150] 11 Processor 100 Biological information estimation device 111, 211 rPPG signal generation unit 112, 122, 132, 212, 222, 232 BNN 121 PPG signal estimation unit 131 Face detection unit 140 Integration unit 213, 223, 233 Optimization unit

Claims

1. A biological information estimation device comprising: acquisition means for acquiring biological images including parts of a subject's skin; and estimation means for estimating predicted values ​​of the subject's biological information and the uncertainty of said predicted values ​​based on the biological images.

2. The biological information estimation device according to claim 1, wherein the estimation means estimates a pulse wave from the biological image and estimates the predicted value and the uncertainty using the estimated pulse wave.

3. The bio-information estimation device according to claim 2, wherein the estimation means detects a face image from the bio-image and estimates the predicted value and the uncertainty using the detected face image.

4. The biological information estimation device according to claim 1, comprising: a first estimation means for calculating a first predicted mean and a first predicted variance of the biological information using pulse waves estimated from the biological image; a second estimation means for calculating a second predicted mean and a second predicted variance of the biological information using a face image detected from the biological image; a predicted value calculation means for calculating a predicted value of the biological information using the first predicted mean and the second predicted mean; and an uncertainty calculation means for calculating the uncertainty using the first predicted variance and the second predicted variance.

5. The bio-information estimation device according to claim 4, wherein the uncertainty calculation means calculates the uncertainty based on the first predictive variance and the second predictive variance.

6. The bio-information estimation device according to claim 4, wherein the uncertainty calculation means calculates the uncertainty based on the variance of the first predicted mean obtained by multiple samplings and the variance of the second predicted mean obtained by multiple samplings.

7. The bio-information estimation device according to claim 4, wherein the uncertainty calculation means calculates the uncertainty based on the variance between the first predicted mean and the second predicted mean.

8. The biological information estimation device according to claim 4, comprising: the biological image being a facial image of the subject, the biological information being blood pressure, a first optimization means for optimizing a deep learning model constituting the first estimation means based on the first predicted mean and the first predicted variance, and a second optimization means for optimizing a deep learning model constituting the second estimation means based on the second predicted mean and the second predicted variance.

9. A method for estimating biometric information performed by a computer, comprising: acquiring a biometric image including a portion of the subject's skin; and estimating a predicted value of the subject's biometric information and the uncertainty of the predicted value based on the biometric image.

10. A recording medium that contains a program that causes a computer to perform a process to acquire biological images including parts of a subject's skin, and to estimate predicted values ​​of the subject's biological information and the uncertainty of said predicted values ​​based on the biological images.

Images