Heart rate measurement device, heart rate measurement method, learning model generation device, trained model, and program

Abstract

Provided is technology capable of more accurately determining a heartbeat cycle waveform.　This heart rate measurement device acquires a data string (the temporal change Φ(τ) in the phase of a complex signal, the temporal change Ψ(τ) in the norm, etc.) based on a signal related to a reflected wave of a transmission wave for each among multiple (N) spatial positions indicated by multiple combinations Cn(r, θ) (n = 1, ..., N) of the distances r between sensors at multiple spatial positions in the vicinity of one living body and the angles θ indicating the directions from the sensors. The heartbeat measurement device estimates the heartbeat cycle waveform of the one living body by inputting, to a learning model 410, multiple (N) data strings based on multiple signals related to multiple (N) combinations (multiple spatial positions). The learning model 410 is a model that has been subjected to machine learning so as to output a heartbeat cycle waveform when multiple (N) data strings based on multiple signals are input.

Description

Heart rate measurement device, heart rate measurement method, learning model generation device, learned model, and program

[0001] The present invention relates to a heart rate measurement device for measuring the heart rate (heartbeat) of a living body and related technologies.

[0002] There is a technology for irradiating a living body with electromagnetic waves for measurement (electromagnetic waves with short wavelengths such as microwaves or millimeter waves), and acquiring vital information (information related to respiration and / or heart rate) of the living body based on the reflected wave reflected by the living body (see Patent Document 1, etc.).

[0003] In Patent Document 1, a complex signal composed of an in-phase signal (I signal) and a quadrature signal (Q signal) of the reflected wave of microwaves transmitted toward a plurality of persons (living bodies) is acquired, and a plurality of vital information of the plurality of persons is acquired (measured) based on the complex signal. Specifically, the position of each person is acquired (one (one location)) based on the complex signal, and vital information (information related to heart rate, etc.) of each person is acquired based on the complex signal at the one location of each person.

[0004] In Patent Document 1, although the heart rate among the information related to the heart rate is acquired (calculated), the heart rate cycle waveform is not acquired.

[0005] Further, in Patent Document 2, it is shown that a heart rate signal is demodulated based on a signal waveform extracted by a frequency filter (such as a band-pass filter) from a reflected wave (also referred to as a reflected signal or a radar signal) of radio waves transmitted to a living body, which includes a respiration signal and a heart rate signal. Specifically, it is shown that envelope detection (such as Hilbert transform) of the filtered signal waveform is performed, and the original heart rate signal is demodulated.

[0006] Japanese Patent No. 7433554 Japanese Unexamined Patent Application Publication No. 2024-5820

[0007] In the technologies of Patent Document 1 and Patent Document 2 described above, a biological signal at a specific location (a specific one location) of each person (human body) is extracted based on the reflected wave from the specific location, and information related to the heart rate is extracted based on the biological signal.

[0008] Such technologies require the appropriate selection (designation) of a specific part of the human body (a single specific location).

[0009] However, there are situations where it is not always easy to appropriately select a specific location. If an inappropriate location is selected, it becomes difficult to accurately determine the heart cycle waveform.

[0010] Therefore, the first objective of this invention is to provide a technology that enables more accurate determination of the heart rate cycle waveform.

[0011] Furthermore, in the technology described in Patent Document 2, the original heartbeat signal is demodulated by determining the envelope.

[0012] However, finding the envelope is not always easy.

[0013] Therefore, our second objective is to provide a technology that allows for easier determination of heart rate cycle waveforms.

[0014] To solve the first problem described above, the heart rate measurement device according to the present invention comprises a control means for estimating the heart rate cycle waveform of a single living organism by acquiring a signal relating to the reflected wave of the transmitted wave and a data sequence based on the signal for each of a plurality of spatial positions near a single living organism, which are represented by a plurality of combinations of the distance between the spatial position and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor, and inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the heart rate cycle waveform when the plurality of data sequences based on the plurality of signals are input.

[0015] The signal relating to the reflected wave is a complex signal composed of an in-phase signal and an orthogonal signal of the reflected wave, and the data sequence input to the learning model may be a data sequence showing the time change in the phase of the complex signal.

[0016] The signal relating to the reflected wave is a complex signal composed of an in-phase signal and an orthogonal signal of the reflected wave, and the data sequence input to the learning model may be a data sequence showing the time evolution of the norm of the complex signal.

[0017] In the learning model described above, a convolution operation may be performed on each of the multiple data sequences, involving two or more data sequences that represent the time change in the phase of the complex signal at the same spatial position.

[0018] In the learning model described above, a convolution operation may be performed on each of the multiple data sequences to generate two or more data sequences that represent the time change in the phase of the complex signal at the same spatial position.

[0019] The signal relating to the reflected wave is a complex signal composed of an in-phase signal and an orthogonal signal of the reflected wave, and the data sequence input to the learning model has a data sequence showing the time change of the phase of the complex signal and a data sequence showing the time change of the norm of the complex signal, and in the learning model, a convolution operation may be performed on each of the plurality of data sequences with respect to the data sequence showing the time change of the phase of the complex signal at the same spatial position and the data sequence showing the time change of the norm of the complex signal at the same spatial position.

[0020] In the convolution process for each of the aforementioned plurality of data sequences, a convolution process may be performed on two or more data sequences that show the time change of the phase of the complex signal at the same spatial position to generate a convolutional phase data sequence which is a data sequence with two or more channels. A convolution process may also be performed on two or more data sequences that show the time change of the norm of the complex signal at the same spatial position to generate a convolutional norm data sequence which is a data sequence with two or more channels. Furthermore, a convolution process may be performed on a data sequence with four or more channels which is an integrated convolutional phase data sequence and a convolutional norm data sequence.

[0021] The learning model is a deep neural network model and comprises an encoder with downsampling and a decoder with upsampling. A new set of data sequences generated by feature extraction through a convolution process on each of the multiple data sequences may be used as the multiple channel data input to the encoder.

[0022] The data sequence input to the learning model may also be a data sequence representing a provisional heart rate period waveform, which is a provisional heart rate period waveform based on an oscillatory wave mainly containing harmonic components of the heartbeat extracted from the signal.

[0023] The learning model may be a deep neural network model comprising an encoder with downsampling and a decoder with upsampling.

[0024] The learning model is a deep neural network model, and the deep neural network model may be a diffusion model.

[0025] The plurality of spatial positions indicated by the plurality of combinations of angles and distances may be positions within a predetermined region that is set in advance as a region in which a part of the one living organism may be included.

[0026] The plurality of spatial positions represented by the plurality of combinations of angle and distance may include positions having a signal strength of a predetermined degree or higher.

[0027] The control means may detect multiple peaks in the heart rate cycle waveform estimated by the learning model and determine the heart rate based on the multiple peaks.

[0028] Each of the plurality of data sequences based on the plurality of signals is data for a predetermined period length, the learning model is a model that has been trained to output the heart rate cycle waveform when the plurality of data sequences over the predetermined period length are input, and the control means may input the plurality of data sequences over the predetermined period length to the learning model for a plurality of different periods, each having the predetermined period length, by sliding the start time of the period by a predetermined minute by a predetermined minute by a certain amount of time, to the learning model and obtain a provisional output waveform which is the heart rate cycle waveform output from the learning model, and generate an integrated heart rate cycle waveform by synthesizing the plurality of provisional output waveforms obtained for the plurality of periods.

[0029] Each of the plurality of data sequences based on the plurality of signals is data for a predetermined period length, the learning model is a model trained to output the heart rate cycle waveform when the plurality of data sequences over the predetermined period length are input, and the control means may input the plurality of data sequences over the predetermined period length to the learning model for each of the plurality of different periods, each having the predetermined period length, by sliding the start time of the period by a predetermined minute time, and obtain a provisional output waveform which is the heart rate cycle waveform output from the learning model, and estimate the peak position of the heart rate cycle waveform of the one living organism based on the peak position of each provisional output waveform obtained for each of the plurality of periods.

[0030] Each input data during training of the learning model is data having a structure in which data sequences relating to each signal of the plurality of combinations are arranged in channels on a data sequence basis, and the learning model may be a machine learning model using a plurality of input data in which the channel arrangement of the plurality of data sequences relating to the plurality of combinations is changed, and the plurality of input data includes a first input data in which the plurality of data sequences are arranged in a first arrangement in a plurality of channels, and a second input data in which the plurality of data sequences are arranged in a second arrangement in a plurality of channels.

[0031] To solve the first problem described above, the heart rate measurement method according to the present invention comprises the steps of: a) acquiring a signal relating to the reflected wave of a transmitted wave and acquiring a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor; and b) estimating the heart rate cycle waveform of the single living organism by inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a model that has been trained to output the heart rate cycle waveform when it is input the plurality of data sequences based on the plurality of signals.

[0032] To solve the first problem described above, the present invention provides a program that causes a computer to perform the following steps: a) acquire a signal relating to the reflected wave of a transmitted wave and acquire a data sequence based on the signal for each of a plurality of spatial locations near a living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor; and b) input a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model to estimate the heart rate cycle waveform of the living organism, wherein the learning model is a machine learning model that outputs the heart rate cycle waveform when it is input to a plurality of data sequences based on the plurality of signals.

[0033] To solve the first problem described above, the learning model generation device according to the present invention includes a control means for machine learning a learning model that takes as input a plurality of data sequences relating to a plurality of spatial positions near a living organism, which are indicated by a plurality of combinations of the distance between the spatial position and a sensor that transmits a transmission wave and an angle indicating the direction from the sensor, and outputs a heart rate cycle waveform, wherein each of the plurality of data sequences is a data sequence based on a signal relating to the reflected wave of the transmission wave, and the control means machine learning the learning model based on training data comprising the plurality of data sequences and the correct data of the heart rate cycle waveform.

[0034] The control means may perform machine learning on the learning model not only based on first training data comprising the plurality of data sequences and the correct data of the heart rate cycle waveform, but also on second training data comprising modified plurality of data sequences obtained by changing the time scale of the plurality of data sequences by a predetermined percentage, and modified correct data obtained by changing the time scale of the correct data by the predetermined percentage.

[0035] Each input data during training of the learning model is data having a structure in which data sequences relating to each of the multiple combinations of signals are arranged in channels on a data sequence basis, and the control means may perform machine learning on the learning model using a plurality of input data in which the channel arrangement of the plurality of data sequences relating to the plurality of combinations is changed, the plurality of input data including a first input data in which the plurality of data sequences are arranged in a first arrangement in a plurality of channels, and a second input data in which the plurality of data sequences are arranged in a second arrangement in a plurality of channels.

[0036] Each input data during training of the learning model is data having a structure in which data sequences relating to each of the multiple combinations of signals are arranged in channels on a data sequence basis, and the control means may also use data in which some of the multiple data sequences relating to the multiple combinations have been replaced with zeros as training data to train the learning model.

[0037] To solve the first problem described above, the trained model according to the present invention is a trained model for causing a computer to function to output a heart rate cycle waveform when it is input a plurality of data sequences relating to a plurality of spatial locations near a living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor, wherein each of the plurality of data sequences is a data sequence relating to the reflected wave of the transmitted wave, and the trained model is a model that has been machine-learned based on training data comprising the plurality of data sequences for learning and ground truth data of the heart rate cycle waveform.

[0038] To solve the first problem described above, the heart rate measurement device according to the present invention comprises a control means for estimating the optimal position among the multiple spatial positions from which the best data regarding the heart rate of the single living organism can be obtained, by acquiring a signal relating to the reflected wave of the transmitted wave and a data sequence based on the signal for each of the multiple spatial positions, which are multiple spatial positions near a single living organism and are represented by multiple combinations of the distance between the sensor transmitting the transmitted wave and the angle indicating the direction from the sensor, and inputting the multiple data sequences based on the multiple signals relating to the multiple combinations into a learning model, wherein the learning model is a machine learning model that outputs the optimal position when the multiple data sequences based on the multiple signals are input.

[0039] To solve the first problem described above, the heart rate measurement method according to the present invention comprises the steps of: a) acquiring a signal relating to the reflected wave of a transmitted wave and acquiring a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor; and b) estimating the optimal position among the plurality of spatial locations from which the best data relating to the heart rate of the single living organism can be obtained by inputting the plurality of data sequences based on the plurality of signals for the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the optimal position when the plurality of data sequences based on the plurality of signals is input.

[0040] To solve the first problem described above, the present invention provides a program that causes a computer to perform the following steps: a) acquire a signal relating to the reflected wave of a transmitted wave and acquire a data sequence based on the signal for each of a plurality of spatial locations near a living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor; and b) input a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model to estimate the optimal location among the plurality of spatial locations from which the best data relating to the heart rate of the living organism can be obtained, wherein the learning model is a machine learning model that outputs the optimal location when it receives the plurality of data sequences based on the plurality of signals.

[0041] To solve the first problem described above, the learning model generation device according to the present invention includes a control means that takes as input a plurality of data sequences relating to a plurality of spatial positions near a single living organism, which are represented by a plurality of combinations of the distance between the spatial position and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor, and outputs a learning model in which the optimal position among the plurality of spatial positions from which the best data relating to the heartbeat of the single living organism can be obtained, wherein each of the plurality of data sequences is a data sequence based on a signal relating to the reflected wave of the transmitted wave, and the control means performs machine learning on the learning model based on training data comprising the plurality of data sequences and the correct data of the optimal position.

[0042] The learning model is a model that has been trained to output not only the optimal position but also the heart rate cycle waveform, and the control means may train the learning model based on training data that includes the plurality of data sequences, the correct data for the optimal position, and the correct data for the heart rate cycle waveform.

[0043] To solve the above first problem, the learned model according to the present invention, when inputting a plurality of data sequences regarding a plurality of spatial positions near a living body, which are represented by a plurality of combinations of the distance between the sensor that transmits the transmission wave and the angle indicating the direction from the sensor, outputs an optimal position capable of acquiring the best data regarding the heartbeat of the living body among the plurality of spatial positions. It is a learned model for causing a computer to function, and each of the plurality of data sequences is a data sequence based on a signal regarding the reflected wave of the transmission wave. The learned model is a model machine-learned based on teacher data configured to include the plurality of data sequences for learning and the correct answer data of the optimal position.

[0044] To solve the above second problem, the heartbeat measurement device according to the present invention has control means for acquiring a signal regarding the reflected wave of the transmission wave transmitted toward the living body. The control means applies a high-pass filter to the signal to generate a first data sequence mainly including the harmonic component of the heartbeat of the living body, then generates a second data sequence by applying an absolute value filter to the first data sequence, and applies a median filter to the second data sequence to convert the data at each time point in the second data sequence into the median value within a predetermined period including each time point, thereby generating the heartbeat cycle waveform of the living body.

[0045] To solve the above second problem, the heartbeat measurement method according to the present invention includes: a) a step of acquiring a signal regarding the reflected wave of the transmission wave transmitted toward the living body; b) a step of applying a high-pass filter to the signal to generate a first data sequence mainly including the harmonic component of the heartbeat of the living body; c) a step of generating a second data sequence by applying an absolute value filter to the first data sequence; and d) a step of generating the heartbeat cycle waveform of the living body by applying a median filter to the second data sequence to convert the data at each time point in the second data sequence into the median value within a predetermined period including each time point.

[0046] In order to solve the above second problem, the program according to the present invention includes: a) a step of acquiring a signal related to a reflected wave of a transmitted wave transmitted toward a living body; b) a step of applying a high-pass filter to the signal to generate a first data series mainly including harmonic components of the heartbeat of the living body; c) a step of generating a second data series by applying an absolute value filter to the first data series; and d) a step of generating a heartbeat cycle waveform of the living body by applying a median filter to the second data series to convert the data at each time point in the second data series into a median value within a predetermined period including each time point. The program is for causing a computer to execute these steps.

[0047] In order to solve the above first problem, a heartbeat cycle waveform is estimated by inputting a plurality of data series based on a plurality of signals related to a plurality of spatial positions near a living body into a learning model. According to this, it is possible to obtain a heartbeat cycle waveform more accurately than when obtaining a heartbeat cycle waveform based on a biological signal at only a specific location.

[0048] Also, in order to solve the above first problem, a learning model is obtained that takes, as an input, a plurality of data series based on a plurality of signals related to a plurality of spatial positions near a living body and outputs a heartbeat cycle waveform. According to this, it is possible to obtain a heartbeat cycle waveform more accurately than when obtaining a heartbeat cycle waveform based on a biological signal at only a specific location.

[0049] Also, in order to solve the above first problem, by inputting a plurality of data series based on a plurality of signals related to a plurality of spatial positions near a living body into a learning model, an optimal position capable of acquiring optimal data related to the heartbeat of a living body among the plurality of spatial positions is estimated. According to this, even when obtaining a heartbeat cycle waveform based on a biological signal at only a specific location, it is possible to obtain a heartbeat cycle waveform more accurately by using the biological signal at the optimal position.

[0050] Furthermore, in order to solve the first problem described above, a learning model is obtained that takes as input multiple data sequences based on multiple signals relating to multiple spatial positions near a single living organism, and outputs the optimal position from which the optimal data relating to the heartbeat of one of the multiple spatial positions can be obtained. With this, even when determining the heart rate cycle waveform based on a biological signal from only one specific location, it is possible to determine the heart rate cycle waveform more accurately by using the biological signal from the optimal position.

[0051] To solve the second problem described above, a first data sequence is generated by applying a high-pass filter to the reflected wave signal to extract the harmonic components of the heartbeat. A second data sequence is generated by applying an absolute value filter to the first data sequence. A median filter is then applied to the second data sequence to convert the data at each point in time in the second data sequence to the median value within a predetermined period including each point in time, thereby generating the heartbeat cycle waveform. This makes it possible to obtain the heartbeat cycle waveform more easily than when using the envelope.

[0052] This is a functional block diagram showing the schematic configuration of a heart rate measurement device. This diagram explains the principle of signals received by virtual array elements. This diagram shows multiple IQ data. This diagram shows an example of millimeter-wave sensor arrangement. This is a conceptual diagram showing a person as seen from a millimeter-wave sensor. This diagram shows the measurement target area, etc. This diagram explains static clutter removal. This diagram shows complex signals before and after applying a high-pass filter. This diagram shows the measurement target area, etc., related to a modified example. This diagram shows the processing in the learning stage and the inference stage. This is a flowchart showing the processing in the learning stage. This is a flowchart showing the processing in the inference stage. This diagram shows an hourglass network model. This diagram shows the time change of the phase of a complex signal with respect to a certain spatial position. This diagram shows examples of inputs and outputs to a learning model. This diagram shows multiple peak intervals, etc., obtained based on the heart rate cycle waveform. This diagram explains time scale transformation. This diagram explains channel shuffling. This diagram explains channel drop. This diagram shows an hourglass network model with confidence output. This diagram shows a spreading model. This diagram shows a spreading model with confidence output. This diagram shows multiple periods in which the start time is shifted by predetermined minute amounts of time. This diagram shows an integrated heart rate cycle waveform, etc., based on multiple provisional output waveforms. This figure shows the integrated peak position, etc., based on multiple provisional output waveforms. This flowchart shows the processing of the fourth embodiment. This figure shows the signals, etc., before and after processing of the fourth embodiment. This figure shows the processing results (estimated heart rate cycle waveform), etc., according to the fifth embodiment. This figure shows the processing of the learning stage and the inference stage of the sixth embodiment. This figure shows the learning model in the sixth embodiment. This figure shows N heart rate cycle waveforms, etc., based on the method of the fourth embodiment. This figure explains the method for obtaining the correct data for the optimal position in the sixth embodiment. This figure shows the processing of the learning stage and the inference stage of the seventh embodiment. This figure shows the learning model in the seventh embodiment. This figure shows the measurement target area, etc., in the eighth embodiment, etc. This figure shows the learning model according to the eighth embodiment. This figure shows a learning model related to a modified version of the eighth embodiment. This figure shows a learning model related to another modified version of the eighth embodiment. This figure shows the learning model according to the ninth embodiment. This figure shows a learning model related to a modified version of the ninth embodiment.This figure shows a learning model relating to another modification of the ninth embodiment. This figure shows a learning model relating to yet another modification of the ninth embodiment.

[0053] Embodiments of the present invention will be described below with reference to the drawings.

[0054] <1. First Embodiment> <1-1. Overview> Figure 1 is a functional block diagram showing the schematic configuration of the heart rate measurement device 10. The heart rate measurement device 10 is a device for measuring the heart rate of a living organism (human or animal).

[0055] The heart rate measurement device 10 acquires a signal (complex signal, etc.) relating to the reflected wave of a transmitted wave (e.g., millimeter wave) sent toward a living organism (e.g., a target organism), and acquires a data sequence based on the signal. The heart rate measurement device 10 acquires a data sequence based on the signal for each of a plurality of spatial positions (in particular, a plurality of spatial positions near a living organism) represented by a plurality (N) combination of distance r and angle θ Cn(r, θ) (n = 1, ..., N; N is an integer of 2 or more). In other words, the heart rate measurement device 10 acquires the signal relating to the reflected wave separately for each distance r and each angle θ (spatial position). Here, distance r is the distance between each spatial position and the sensor (millimeter wave sensor 20, etc.) that transmits the transmitted wave. In other words, distance r is the distance (from each spatial position) to the sensor that transmits the transmitted wave (which is also the sensor that receives the reflected wave). Furthermore, angle θ represents the direction from the sensor (the direction as seen from the sensor), and more specifically, it represents the direction of each spatial position relative to the sensor's reference direction (laser line of sight direction). Angle θ can also be expressed as the angle representing the direction of arrival of the reflected wave.

[0056] In this embodiment, the heart rate measurement device 10 acquires a complex signal as a signal related to the reflected wave, which is composed of an in-phase signal and an orthogonal signal of the reflected wave, and acquires a data sequence based on the complex signal. Each data sequence is a data sequence that shows the time change of the complex signal, for example, the time change in the phase of the complex signal Φn(τ) is acquired. The distance r can also be expressed as the distance between the millimeter-wave sensor 20 and the virtual reflection position of the reflected wave (the spatial position where the transmitted wave is considered to have been virtually reflected). Furthermore, as will be described later, the time change in the norm Ψ(τ) of the complex signal may be used as a data sequence that shows the time change of the complex signal, and signals other than complex signals may be used as signals related to the reflected wave.

[0057] The heart rate measurement device 10 then inputs multiple (N) data sequences (for example, Φn(τ)) based on multiple (N) signals (complex signals, etc.) relating to multiple (N) combinations Cn (n=1,...,N) (N spatial positions) into a learning model 410 (see also Figure 10) to estimate (measure) the heart rate cycle waveform of the living organism (e.g., the person being estimated).

[0058] The learning model 410 is a model that has been pre-trained to output a heart rate cycle waveform (such as a periodic oscillation waveform of the heartbeat) when it is input with multiple (N) data sequences based on multiple (N) signals. A deep neural network model or the like can be used as the learning model 410. The learning model 410 after being trained by machine learning is also called the trained model 420.

[0059] Furthermore, the transmitted waves are not limited to millimeter waves; microwaves or other types of waves may also be used.

[0060] <1-2. Detailed Configuration of Heart Rate Measurement Device 10> The heart rate measurement device 10 is a device that estimates (measures) the heart rate cycle waveform, etc., of a person being measured based on measurement data (in this embodiment, measurement data from the millimeter-wave sensor 20), etc. The heart rate measurement device 10 is a device that measures information related to the heart's beating (heart rate), etc. The heart rate measurement device 10 does not necessarily need to determine the heart rate, and may be a device that only determines the heart rate cycle waveform (heart rate waveform). The heart rate measurement device 10 is also called a heart rate waveform measurement device.

[0061] As shown in Figure 1, the heart rate measurement device 10 comprises a millimeter-wave sensor 20 and a processing unit 30.

[0062] The millimeter-wave sensor 20 transmits radio waves in the millimeter-wave band (transmitted waves) and receives radio waves (received waves) reflected from an object. The millimeter-wave sensor 20 is a sensor (radar sensor) capable of measuring the distance r from the millimeter-wave sensor 20 to an object (distance r to the virtual reflection position), and the azimuth angle θ of the object (arrival angle of the reflected wave) based on the transmitted wave and the received wave (also called the reflected wave). Here, only the angle θ around a predetermined axis is measured as the azimuth angle. However, it is not limited to this, and two azimuth angles (θ, ψ) in two different directions (orthogonal directions) may be measured as the azimuth angle.

[0063] Furthermore, the millimeter-wave sensor 20 can acquire signals related to reflected waves separately for each distance r and each angle θ(ψ) (spatial position). In other words, the millimeter-wave sensor 20 can acquire signals related to reflected waves for each of the multiple spatial positions represented by multiple combinations Cn(r,θ) (or Cn(r,θ,ψ) etc.) (hereinafter also simply referred to as Cn), and can also acquire data sequences (Φn(τ) etc. (τ is time)) based on said signals. The millimeter-wave sensor 20 can then estimate the heart rate cycle waveform based on the data sequences based on said signals. More specifically, the millimeter-wave sensor 20 can estimate the heart rate cycle waveform by inputting multiple data sequences based on multiple signals related to the multiple combinations Cn(r,θ) etc. into a learning model (DNN). Examples of data sequences related to combination Cn include data sequences showing the time evolution of each signal, and more specifically, the time evolution of the phase of the complex signal related to combination Cn (Φn(τ)).

[0064] The millimeter-wave sensor 20 is installed, for example, in a position where the person being measured is viewed from above (for example, on the ceiling of a living room (or store) or on a side wall near the ceiling), and is set to have a downward angle. The millimeter-wave sensor 20 measures the heart rate (heart rate cycle waveform, etc.) of a person in the space being measured. Specifically, the millimeter-wave sensor 20 acquires minute fluctuations of multiple minute parts on the surface of the person (human body) in a time series, and measures the person's heart rate cycle waveform, etc., based on these minute fluctuations.

[0065] The millimeter-wave sensor 20 incorporates an antenna unit 20a that transmits a wave and receives a wave (reflected wave), and an internal controller 20b that performs transmission and reception processing of the transmitted and received waves, as well as analysis processing of both waves, etc., (within the housing of the millimeter-wave sensor 20).

[0066] The millimeter-wave sensor 20 may measure the heart rate of a person in the measurement target space by calculation processing using only the internal controller 20b. However, in this embodiment, the millimeter-wave sensor 20 works in cooperation with an external controller (for example, the controller 31 of the processing unit 30 (described later)) to measure the person's heart rate based on the data acquired by the millimeter-wave sensor 20 (measurement data from the millimeter-wave sensor 20 (such as the time change of the phase of the complex signal)). Specifically, the measurement data from the millimeter-wave sensor 20 is input to the controller 31 of the processing unit 30, and the person's heart rate cycle waveform, etc., is ultimately measured (estimated) by calculation processing by the controller 31.

[0067] Furthermore, the processing unit 30 includes a controller (also called a control unit) 31, a storage unit 32, a communication unit 34, and an operation unit 35 (Figure 1).

[0068] The controller 31 is a control device built into the processing unit 30 that controls the heart rate measurement device 10.

[0069] The controller 31 is configured as a computer system equipped with one or more hardware processors (for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)). The controller 31 performs various processes by executing a predetermined software program (hereinafter also simply referred to as "program") stored in a storage unit (ROM and / or a non-volatile storage unit such as a hard disk) 32 using the CPU, etc. The program (more specifically, a group of program modules) (also referred to as "program product") may be recorded on a portable recording medium such as a USB memory stick, read from the recording medium, and installed in the heart rate measurement device 10. Alternatively, the program may be downloaded via a communication network or the like and installed in the heart rate measurement device 10.

[0070] The controller 31 recognizes the person's heart rate, etc., based on the data measured (acquired) by the millimeter-wave sensor 20.

[0071] The memory unit (also called the storage unit) 32 is composed of a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The memory unit 32 stores (stores) time-series data of signals related to the reflected waves of transmitted waves sent towards a person.

[0072] The operation unit 35 includes an operation input unit 35a that receives operation input for the heart rate measurement device 10, and a display unit 35b that displays and outputs various information.

[0073] The communication unit 34 is capable of performing network communication via the network 108. Various protocols, such as TCP / IP (Transmission Control Protocol / Internet Protocol), can be used in this network communication. By utilizing this network communication, the heart rate measurement device 10 can exchange various types of data with a desired partner. For example, it is possible to transmit heart rate cycle waveform data to an external device (external terminal device) separate from the heart rate measurement device 10, and to display the heart rate cycle waveform on the external terminal based on that data.

[0074] <1-3. Signals related to transmitted and reflected (received) waves> <Complex signals for each combination Cn(r,θ)> The antenna section 20a of the millimeter-wave sensor 20 employs a MIMO (Multiple-Input and Multiple-Output) method, which uses multiple antennas for transmission and reception. In other words, a MIMO-type millimeter-wave sensor is used as the millimeter-wave sensor 20. In the MIMO method, a number of virtual array elements (virtual array antennas composed of multiple virtual array elements (virtual antennas)) is virtually operated, equal to the product of the number of physical transmitting antennas 22 (see Figure 2) and the number of physical receiving antennas 23. For example, if the number of transmitting antennas is 3 and the number of receiving antennas is 4, it can be considered that 12 (= 3 × 4) virtual array elements (virtual antennas) are installed. Figure 2 is a diagram that explains in principle the signals received by the 12 virtual array elements. In Figure 2, one transmitting antenna 22 and one receiving antenna 23 are shown, but this differs from the actual number (for example, three and four).

[0075] The millimeter-wave sensor 20 transmits a modulated wave (also called a chirp) using, for example, FMCW (Continuous Frequency Modulation) as the transmission wave toward the space to be measured. This transmission wave is generated by the synthesizer (oscillator) 21. The transmission wave from the millimeter-wave sensor 20 is reflected by objects (such as living organisms) in the space to be measured, and the reflected wave is received by the millimeter-wave sensor 20 (more specifically, by multiple virtual array elements). The millimeter-wave sensor 20 acquires the radio waves received by each of the multiple virtual array elements as complex signals (also called IQ signals or IQ data). Specifically, the received wave (received signal) and the transmitted wave (transmitted signal) are combined by a mixer to generate an intermediate frequency signal. More specifically, the reflected wave (received wave) received by the virtual array elements is mixed with the in-phase transmitted wave by the mixer 24a and output as an in-phase signal (I signal) SI. Furthermore, the reflected wave (received wave) received by the virtual array element is mixed with the transmitted wave, which has been shifted by 90 degrees in phase, by the mixer 24b and output as an orthogonal signal (Q signal) SQ. Then, a complex signal S = SI + jSQ (where j is the imaginary unit) is generated based on the common-mode signal (I signal) SI and the orthogonal signal (Q signal) SQ. This complex signal is sampled at high speed (with a very small sampling interval Δt (for example, about 0.8 μs (microseconds))) and acquired as digital data. In detail, the common-mode signal SI and the orthogonal signal SQ are each converted using separate A / D converters. As a result, multiple (number of samples (for example, 256)) IQ data are generated corresponding to each virtual array element (virtual antenna).

[0076] More specifically, first, transmission and reception processing is performed for one chirp set (a unit of radio wave transmission by three physical transmitting antennas 22 (a unit of reception processing in 12 virtual array elements)). In detail, the process of simultaneously receiving the reflected wave from one physical transmitting antenna 22 at four physical receiving antennas 23 is repeated three times (for the three physical transmitting antennas 22). This executes reception processing at 12 virtual array elements (in other words, one chirp set process). One chirp set process is executed within a very short period of time (for example, 0.7 ms (milliseconds)), and the received data at the 12 virtual array elements can be considered as data acquired simultaneously.

[0077] Furthermore, this chirp set processing (reception processing with 12 virtual array elements) is repeatedly performed for further chirp sets at a predetermined sampling interval Δτ (for example, an interval of 10 ms (milliseconds)). More specifically, after the transmission of one chirp set is completed, there is a predetermined period of time before the transmission of the next chirp set is performed, and this is repeated at time intervals Δτ.

[0078] This results in the acquisition of multiple IQ data points, as shown in the upper part of Figure 3. The small cubes in the upper part of Figure 3 represent the IQ data for each virtual array element in units of a small time interval (Δt), in other words, the IQ data for each virtual array element at each time interval t. In the upper part of Figure 3, the left-right direction corresponds to time t, the depth direction corresponds to the virtual array element number, and the downward direction corresponds to the chirpset number (time τ).

[0079] However, the following processing is performed on each chirp set. Therefore, in reality, IQ data like that shown in the lower part of Figure 3 is generated, rather than the upper part. The small cubes in the lower part of Figure 3 (also called radar cubes) show the IQ data for each distance r and angle θ.

[0080] Specifically, first, a Fast Fourier Transform (FFT) is applied to multiple (for example, 256) complex signals (multiple IQ data) in the time series received by each virtual array element. This transforms the data from the time domain to the frequency domain. Since the transformed frequency corresponds to the distance r, the transformation from the time domain to the frequency domain is followed by a transformation to the distance domain (distance r). In short, IQ data is obtained for each distance r. As a result, IQ data (also called IQ data after distance FFT) is obtained for each interval (distance interval) divided into small distance units (for example, distance resolution units). Note that when the distance FFT is applied to a complex signal (IQ data), it remains a complex signal (IQ data) after the application.

[0081] In this way, IQ data for each distance (IQ data after distance FFT) is obtained for each of the multiple virtual array elements. Note that the distance axis (r axis) in the lower part of Figure 3 is the axis obtained by converting the time axis of each virtual array element via the conversion to the frequency axis. Also, the curve L1 in the lower part of Figure 3 shows an example of the signal strength after distance conversion processing of a certain virtual array element.

[0082] Furthermore, by applying the beamformer method (or the CAPON method, etc.) to the IQ data for the same distance r (same distance bin) among the multiple IQ data (IQ data after distance FFT) for multiple virtual array elements, the direction of arrival of the reflected wave (angle θ) can be determined. This allows for obtaining IQ data for each interval (angle interval) divided into small angular units (e.g., angular resolution units). In short, IQ data for each small angle can be obtained. Note that the axis showing the depth direction in the lower part of Figure 3 corresponds to the angle θ transformed based on the signals of multiple virtual array elements (also called the azimuth axis).

[0083] As a result, IQ data is obtained for each distance r and angle θ (see Figure 3, bottom). In other words, IQ data is obtained for each combination of distance r and angle θ. To put it another way, for each of the multiple combinations Cn(r, θ) of angle θ and distance r, a complex signal consisting of the in-phase signal and the orthogonal signal of the reflected wave is acquired.

[0084] This process is first performed on one chirp set (a unit of radio wave transmission by three physical transmitting antennas 22 (a unit of reception processing by 12 virtual array elements)). Then, this process is repeated for multiple chirp sets at predetermined small time intervals Δτ (for example, 10 ms (milliseconds)), thereby obtaining the time variation of the complex signal (IQ data) (IQ data for each time τ) for each of the multiple combinations Cn(r, θ) of distance r and angle θ. In short, the time variation of the IQ data at each spatial position is obtained. The time axis τ in the lower part of Figure 3 corresponds to the direction of the number of chirp sets.

[0085] Here, the combination of distance r and angle θ Cn(r, θ) corresponds to a certain spatial position within the measurement target space (also expressed as a virtual reflection position of the reflected wave). The time change of the signal (complex signal) for each combination Cn(r, θ) indicates the change at each position within the measurement target space, for example, minute skin displacements (time displacements) at each position such as the chest of the human body. Each combination Cn(r, θ) can correspond to a position on the human body surface, a position inside the human body, and a position outside the human body. Strictly speaking, the signal for a position outside the human body surface does not represent the skin displacement of the human body itself. However, in this embodiment, considering that the signal for a position outside the human body surface (such as a position inside the human body and a position close to the human body surface) is affected by the skin displacement at the human body surface, the signal for a position outside the human body surface is also used. Hereafter, the combination Cn(r, θ) will also be referred to as the spatial position Cn(r, θ).

[0086] Figure 4 shows an example of the arrangement of the millimeter-wave sensor 20, in other words, an example of the positional relationship between the millimeter-wave sensor 20 and the person being measured (viewed from the side of the human body). Figure 4 shows a situation in which the heart rate of a person in an upright position (standing person) is measured by the millimeter-wave sensor 20 set diagonally above the person. In detail, it shows how a standing person is measured at a position 80 cm horizontally away from directly below the millimeter-wave sensor 20. Although a situation in which a person in an upright position is measured is shown as an example, it is not limited to this, and a person in other positions (such as lying down) may also be measured.

[0087] Here, as shown in Figure 4, the millimeter-wave sensor 20 is installed so that its radar line of sight is directed towards the vicinity of the abdomen of the human body. For example, if angle θ is the angle of deviation from the line of sight (radar line of sight) of the millimeter-wave sensor 20 (millimeter-wave radar), then the measurable range (range of measurable angles) of the millimeter-wave sensor 20 is set to an angle θ in the range of +30 degrees (upward) to -30 degrees (downward) with respect to the line of sight. Distance r is the distance from the millimeter-wave sensor 20, and for example, a distance r in the range of 0m to 2m (meters) is set to the measurable range (range of measurable distance) of the millimeter-wave sensor 20. In Figure 4, the two-dimensional measurable range (relating to both angle θ and distance r) is shown in the shaded area. Figure 6 is also a diagram showing the measurable range. However, the horizontal axis in Figure 6 is the distance r axis, and the vertical axis in Figure 6 is the angle θ axis. That is, Figure 6 is a diagram showing the measurable range in the rθ plane (distance r - angle θ plane).

[0088] Figure 5 is a conceptual diagram showing the subject (person to be measured) as seen from the millimeter-wave sensor 20, and corresponds to a view of the person from the front. In Figure 5, the measurable range of the millimeter-wave sensor 20 is shown enclosed by a thick dashed line. Here, a measurable space with a width of approximately 3 degrees in the left-right direction of Figure 5 (2 meters in the depth direction of Figure 5) is assumed.

[0089] When measuring heart rate, it is preferable that IQ data of the human body's position (rather than other parts of the body) is captured well. In particular, it is preferable that minute displacements in the chest area of ​​the human body are captured.

[0090] Therefore, in this embodiment, a predetermined region (predetermined space) 110 (see Figure 6) within the measurable space that may include a part of a living organism (human body) (such as the chest area) is pre-set as the measurement target region (measurement target space). Then, IQ data for multiple locations within the predetermined region 110 is acquired as IQ data for multiple spatial locations near a single living organism. The predetermined region 110 is also referred to as the IQ data extraction target region because it is the region from which the IQ data to be used is extracted.

[0091] As can be seen by referring to Figures 4 and 5, at an angle θ = 30 degrees, the head of the human body is relatively close to the millimeter-wave sensor 20, so the distance r to the head is relatively small. On the other hand, as the angle θ decreases (as you move towards the lower side of the human body), each part of the human body gradually moves away from the millimeter-wave sensor 20 (the distance r to each part of the human body gradually increases).

[0092] In such cases, as shown in the upper part of Figure 6 (Figure 6(a)), a predetermined region 110 (110a) is set in advance as the measurement target region. The predetermined region 110a is composed of multiple (N; here, N = 192) divided regions (rectangular regions) that are divided into, for example, 24 × 8 (192 in total) (divided into 24 in the angle θ direction and 8 in the distance r direction). When the angular resolution is approximately 2 degrees and the distance resolution is approximately 5 cm, the predetermined region 110a is a region that spans an angular range of approximately 48 degrees in the θ direction and a distance range of approximately 40 cm (centimeters) in the r direction. In other words, the predetermined region 110a is set to span an angular range of +30 degrees to approximately -18 degrees and a distance range of approximately 100 cm (centimeters) to approximately 140 cm (centimeters) in the distance r. Note that in Figure 6, the region where the human body is located is roughly shown by an elliptical dashed line.

[0093] This predetermined region 110 is set in advance as a region that includes parts of a living organism (human body) (especially the chest, etc.) (a region that is highly likely to include parts of the human body). Then, complex signals (IQ data) acquired at each of multiple (N locations) (in this case, N = 192 locations) within this predetermined region 110 are used. In this way, complex signals of multiple spatial locations (multiple combinations) Cn(r, θ) within the predetermined region 110, which is set in advance as a region that may include parts of the human body, are used.

[0094] The multiple (192) locations include positions on the human body (positions on the body surface and positions inside the body) and positions other than those on the human body. By including as many positions on the human body as possible among the multiple (192) locations, the heart rate cycle waveform of the person in question can be measured accurately.

[0095] The predetermined region 110 is preferably changed according to the position of the human body being measured. For example, when measuring a human body standing at a position 40 cm horizontally away from the position directly below the millimeter-wave sensor 20, a predetermined region 110 (also referred to as 110b) as shown in Figure 6(b) should be set in advance. The predetermined region 110b is a region set at a position closer to the millimeter-wave sensor 20 compared to the predetermined region 110a.

[0096] Furthermore, if the position of the human body (where the person is standing) changes within the range of "40 cm" to "80 cm", a new predetermined region including both the aforementioned predetermined regions 110a and 110b may be set as the measurement target region. In such cases, an appropriate value N (a relatively large value N (for example, N = 256)) may be adopted.

[0097] In Figure 6, a rectangular region (a rectangular region where the minimum distance r is the same for each angle θ) within the rθ plane (distance r - angle θ plane) is set as the predetermined region 110 (measurement target region), but the figure is not limited to this.

[0098] For example, a parallelogram region (see Figure 9) within the rθ plane (distance r - angle θ plane) may be set as the predetermined region 110. More specifically, as shown in Figure 9, a parallelogram region where the minimum distance r increases as the angle θ decreases may be set as the predetermined region 110. In other words, a region with a shape adjusted (deformed) to include a large portion of the area where the human body exists may be set as the predetermined region 110.

[0099] Figure 9(a) shows a predetermined parallelogram-shaped region 110 (110c) when measuring a standing human body at a position 80 cm horizontally away from the position directly below the millimeter-wave sensor 20. Figure 9(b) shows a predetermined parallelogram-shaped region 110 (110d) when measuring a standing human body at a position 40 cm horizontally away from the position directly below the millimeter-wave sensor 20.

[0100] <Removal of Stationary Clutter Components> As described above, signals (IQ data) are acquired at each spatial location. These signals (vibration waves) include vibration components of respiration and heartbeat, as well as components due to stationary clutter (reflected components from walls and floors, etc.). The vibration components related to respiration and heartbeat change periodically and perform periodic motion, moving between clockwise and counterclockwise directions along an arc centered on a virtual origin, synchronized with the person's breathing, etc. On the other hand, the component due to stationary clutter is due to the influence of objects fixed in the environment and is a fixed component (DC component).

[0101] Thus, the complex signal (IQ data) for each combination Cn(r, θ) contains both oscillatory components related to respiration and heartbeat, and components due to static clutter. Therefore, as shown in the upper part of Figure 7 (Figure 7(a)), the oscillation center (virtual center point of periodic motion) of the complex signal (IQ data) for each combination Cn(r, θ) is located at a position shifted from the origin. Figure 7 is a plot of the complex signal (IQ data) for a certain combination Cn (a certain spatial position) on the complex plane. The upper part of Figure 7 (Figure 7(a)) is the plot before the effect of static clutter is removed.

[0102] Therefore, in this embodiment, the position of a virtual origin (virtual center), which is the center point of the periodic motion of the complex signal on the complex plane, is estimated, and the component from the origin to the virtual origin (see the dashed arrow in the upper part of Figure 7) is removed as a stationary clutter component (DC component). The virtual origin can be determined by circle fitting with respect to multiple plot positions. Alternatively, the centroid position of multiple plot positions (in other words, the average value of the complex signal over a certain period) may be determined as the virtual origin. Then, the deviation of the virtual origin (virtual center) from the origin should be removed as the stationary clutter component.

[0103] As a result, as shown in the lower part of Figure 7 (Figure 7(b)), the oscillation components related to respiration and heart rate are represented as the trajectory of periodic motion with the virtual origin moved to the origin position. Note that the lower part of Figure 7 (Figure 7(b)) is a plot after removing the effect of stationary clutter.

[0104] Furthermore, after removing the stationary clutter component, the time change in phase Φn(τ) indicates a change in distance between the reflection position (e.g., the chest) and the millimeter-wave sensor 20 (a change in distance smaller than distance r), and the time change in norm Ψ(τ) indicates a change in signal intensity at the reflection position.

[0105] <Extraction of the heartbeat vibration component> The complex signal (vibration wave) from which the static clutter component has been removed in this way contains the vibration component of respiration and the vibration component of heartbeat. Basically, the vibration frequency of heartbeat is higher than the vibration frequency of respiration.

[0106] Therefore, a high-pass filter is used to remove the low-frequency components (respiratory vibration components) from the vibration wave, and the heartbeat vibration components are extracted. However, the high-pass filter used has a cutoff frequency (stoppage frequency) (e.g., 5 Hz) that is higher than the value obtained by adding a margin to the fundamental vibration component of respiration (e.g., 0.3 Hz) (e.g., 0.5 Hz).

[0107] In this context, respiration generates relatively large harmonics with a larger amplitude compared to the heartbeat. Therefore, if the harmonics of respiration (also called respiratory harmonics) overlap with the frequency of the heartbeat, detecting the heartbeat becomes difficult. For example, if the frequency of respiration (fundamental frequency) is 0.3 Hz, the third harmonic (third harmonic) or fourth harmonic (fourth harmonic) of the fundamental frequency of respiration will overlap with the fundamental frequency of the heartbeat (for example, approximately 0.9 Hz or 1.2 Hz). In this case, extracting the vibrations of the heartbeat becomes difficult.

[0108] Therefore, in this embodiment, in order to avoid the influence of such respiratory harmonics, the heart rate is detected based on the harmonic components of the heart rate rather than the fundamental frequency of the heart rate.

[0109] Specifically, a high-pass filter with a relatively high cutoff frequency (e.g., 5 Hz) is used to remove low-frequency components, specifically respiratory components (particularly the fundamental frequency component of respiration and harmonic components up to a certain order (e.g., the 4th or 5th order)), from the original vibrational wave. Then, based on the vibrational wave after this removal (mainly the harmonic components of the heartbeat), a vibrational waveform (heartbeat cycle waveform) including the fundamental frequency component of the heartbeat is reconstructed.

[0110] Figure 8 shows the time evolution of complex signals (complex signals after removing the static clutter component) for two of the N spatial locations. The upper part of Figure 8 (Figure 8(a)) shows the time evolution of the complex signal at a certain spatial location, and the lower part of Figure 8 (Figure 8(b)) shows the time evolution of the complex signal at a different spatial location. The left side of Figure 8 shows the complex signal (after removing the static clutter component and before applying the high-pass filter), and the right side of Figure 8 shows the complex signal (after removing the static clutter component and after applying the high-pass filter). In the four graphs of Figure 8, the solid lines (curves L11, L21) show the time evolution of the phase of the complex signal, and the dashed lines (curves L12, L22) show the time evolution of the norm (signal intensity) of the complex signal. The horizontal axis of Figure 8 represents time τ (more specifically, the chirpset number (also called the frame number)).

[0111] For example, curve L11 in Figure 8 (see graph on the left) shows the time evolution Φ(τ) of the phase Φ of a complex signal that includes both the respiration and heartbeat oscillatory components. Phase unwrapping processing shows a phase change over a range larger than 360 degrees. Curve L12 shows the time evolution Ψ(τ) of the norm Ψ of the same complex signal.

[0112] On the other hand, curve L21 in Figure 8 (see graph on the right) shows the time change Φ(τ) of the phase Φ of the complex signal after removing the respiratory component (including the main high-frequency components of respiratory) using a high-pass filter. Curve L22 shows the time change of the norm of the complex signal after the removal of the high-frequency components.

[0113] By applying a high-pass filter in this way, the time change Φ(τ) (curve L21) of the phase Φ of the complex signal after removing the respiration component (including the main high-frequency component of respiration) for a certain combination Cn(r,θ) can be obtained.

[0114] Similarly, for multiple (N) combinations Cn(r, θ), the time evolution Φ(τ) (curve L21) of the phase Φ of the complex signal (corresponding to the harmonic components of the heartbeat) after removing the respiratory component (including the main high-frequency components of respiration) can be determined.

[0115] In this embodiment, the vibrational waveform (heartbeat cycle waveform) including the fundamental frequency components of the heartbeat is reconstructed not only based on a complex signal for a single combination Cn(r, θ), but also based on complex signals for multiple (N) combinations of Cn(r, θ) (see Figure 15). More specifically, the heartbeat cycle waveform (the curve L51 at the bottom of Figure 15 (described later)) is reconstructed based on a data sequence of complex signals for multiple combinations of Cn(r, θ) after the respiratory component (including the main high-frequency component of respiration) has been removed (for example, a data sequence showing the time change Φ(τ) of the phase Φ of the complex signal).

[0116] It is also possible to reconstruct the vibration waveform (heartbeat cycle waveform) including the fundamental frequency components of the heartbeat based solely on a complex signal relating to a single combination Cn(r, θ). Such embodiments will be described later (in the fourth embodiment, etc.). However, by using complex signals relating to multiple combinations Cn(r, θ), as in this embodiment, it is possible to generate (reconstruct) the heartbeat cycle waveform more accurately.

[0117] <1-4. Learning Model for Estimating Heart Rate Cycle Waveform> In this embodiment, the heart rate cycle waveform is estimated (reconstructed) using the learning model 410. For this reason, the heart rate measurement device 10 is equipped with a learning model 410 (Figure 1) for estimating the human heart rate cycle waveform. Specifically, the learning parameters of the learning model 410 (learner) are adjusted using a predetermined machine learning method, and a trained learning model 410 (trained model 420) is generated (see upper part of Figure 10). Using this trained model 420, the heart rate cycle waveform of the person being measured (subject to measurement) is estimated (inferred) (see lower part of Figure 10). Figure 10 shows the processing at the learning stage and the processing at the inference stage. These processes are performed by the heart rate measurement device 10 (controller 31). The heart rate measurement device 10 is also a device that performs these information processing operations, and is therefore also referred to as an information processing device.

[0118] As the learning model 410, for example, a neural network model (especially a deep neural network model: DNN model) composed of multiple layers is used. Then, weighting coefficients, etc. (learning parameters) for multiple layers (input layer, (one or more) hidden layers, output layer) in the neural network model are adjusted by a predetermined machine learning method. As the neural network model (deep neural network model), for example, an hourglass-type network model 310 (described later) as shown in Figure 13 is used.

[0119] In this embodiment, a set of data sequences as described above (for example, a phase change group: Φn(τ) (n=1,...,N)) is used as input to the learning model 410.

[0120] More specifically, for each of the multiple combinations Cn(r, θ) of distance r and angle θ as described above, a complex signal consisting of the in-phase signal and the orthogonal signal of the reflected wave is acquired, and each data sequence based on each complex signal is acquired. More specifically, the complex signal (IQ data) after the stationary clutter component has been removed and a high-pass filter has been applied is acquired for each of the multiple combinations Cn(r, θ). As each data sequence, a data sequence showing the time evolution of each complex signal is used, for example, a data sequence Φ(τ) showing the time evolution of the phase Φ of each complex signal.

[0121] Such data sequences are obtained for multiple (N) combinations (N spatial positions). Specifically, as N data sequences based on N complex signals for the N combinations Cn(r,θ) (n=1,...,N), for example, a collection of data sequences showing the phase Φ time change of each of the N complex signals (phase change group: Φn(τ) (n=1,...,N)) is obtained.

[0122] A collection of such data sequences (for example, a phase change group: Φn(τ) (n=1,...,N)) (data set) is used as input to the learning model 410. The learning model 410 outputs the heart rate cycle waveform (estimated waveform) of a person in response to this input about the person.

[0123] Thus, the learning model 410 is a model that takes as input multiple (N) data sequences (for example, the time change Φn(τ) (n=1,...,N)) of the phase Φ of a complex signal Cn(r,θ) and outputs a heart rate cycle waveform.

[0124] During the learning phase, the learning model 410 learns the relationship between multiple (N) data sequences related to the human body and the heart rate waveform of that human body. For example, the relationship between N data sequences Φn(τ) (n=1,...,N) measured for each person and the heart rate waveform of each person is learned through machine learning. Specifically, the learning model 410 learns through machine learning based on training data (training data) which consists of these multiple data sequences and ground truth data of heart rate waveforms.

[0125] Furthermore, each of the multiple data sequences based on multiple complex signals represents data for a predetermined period length (in this case, 5 seconds). The learning model 410 is trained to output a heart cycle waveform when it receives multiple (N) data sequences spanning the predetermined period length as input.

[0126] For the correct data of a person's heart rate cycle waveform, heart rate cycle waveforms (also called PPG waveforms) measured using a PPG (Photoplethysmography) sensor (optical heart rate sensor) are used. For example, for a person, N data sequences and the heart rate cycle waveform measured by the PPG sensor are used simultaneously (more specifically, the signal waveform acquired by the sensor is used as is). The PPG sensor is a sensor that measures heart rate by transmitting light through the skin of the earlobe (or fingertip, etc.) and detecting changes in blood flow in the capillaries.

[0127] Alternatively, the heart cycle waveform measured using an ECG (Electrocardiogram) sensor (electrocardiograph) may be used as the ground truth data for the heart cycle waveform.

[0128] Furthermore, instead of using the signal acquired by the sensor as is, a periodic waveform regenerated from only the peak position of the signal (heart rate periodic waveform) may be used as the correct data. (For example, a triangular wave (or a smoothly changing periodic waveform) regenerated using the peak position of the maximum value of the waveform for a PPG sensor, or the peak position of the maximum value of the R wave for an ECG sensor.)

[0129] Then, inference processing is performed using the machine learning model 410 (trained model 420). Specifically, in the inference stage, multiple data sequences measured (acquired) using the millimeter-wave sensor 20 for the person to be estimated are input to the machine learning model 410 (trained model 420), and the heart rate cycle waveform (estimated waveform) of the person to be estimated is output.

[0130] In this embodiment, not only is a data sequence for the complex signal of a single location on the person being measured used, but also data sequences for the complex signals of multiple locations (N locations) on the same person being measured. This allows for the generation of a good heart rate cycle waveform.

[0131] Figure 13 shows an example of an hourglass-shaped network model 310.

[0132] The hourglass-shaped network model 310 in Figure 13 is one type of deep neural network model. The hourglass-shaped network model 310 comprises an input layer 311, an encoder 312 with downsampling (also referred to as a feature extraction unit), a bottleneck layer 313, a decoder 314 with upsampling (also referred to as a demodulation unit or high-resolution unit), and an output layer 315. The hourglass-shaped network model 310 may also have skip connections (such as skip connections that connect feature maps of the same size in the encoder and decoder). U-Net is an example of an hourglass-shaped network model 310.

[0133] In the hourglass-shaped network model 310 shown in Figure 13, the data size of the input layer 311 is 240 (number of data points per channel) × 192 (channels). In other words, the input data 311D to the input layer 311 is composed of N channels (N=192) of data, each channel having 240 data points (elements). Here, the case of N=192 (=24 × 8) is shown as an example, and the time change Φn(τ) (n=1,...,192) of the phase Φ of 192 complex signals related to 192 spatial positions Cn(r,θ) is input. Note that the data size may be changed as appropriate (the number of data points per channel and the number of channels are examples, and other values ​​may be used).

[0134] Each channel's data represents the time variation Φ(τ) of a single phase Φ, and consists of 240 data points. More specifically, each channel's data represents the time variation Φn(τ) of the phase Φ of a complex signal related to a single combination Cn(r,θ), and is data (data converted A / D at a predetermined sampling rate) over a predetermined period (for example, 5 seconds). More specifically, for example, 500 data points obtained by sampling a 5-second signal at a sampling rate of 100 fps (frames per second) (sampling interval Δτ of 10 ms) are converted into 240 data points by linear interpolation to generate one channel's data (see also the top row of Figure 17). Similarly, data representing the time variation Φ(τ) of a different phase Φ is generated as data for each channel, resulting in a total of N channels of data.

[0135] The upper part of Figure 14 (Figure 14(a)) shows an example of the time change Φ(τ) (curve L21) of a single phase Φ over a 30-second period. Of these, as shown in the lower part of Figure 14 (Figure 14(b)), for example, the time change Φ(τ) over the first 5 seconds is used as input data 311D for the learning model 410.

[0136] Thus, the input data 311D (see Figure 13) has a structure (multi-channel structure) in which data sequences (one-dimensional array data consisting of 240 elements) relating to the complex signals of multiple (192) spatial positions Cn(r, θ) are arranged in individual channels on a data sequence basis (for each data sequence).

[0137] The encoder 312 is composed of multiple layers. In each layer, normalization processing (such as batch normalization processing) and convolution processing are performed. Data for the bottleneck layer 313 is generated by repeatedly performing convolution processing (and normalization processing, etc.) on the input data 311D input to the input layer in the encoder 312.

[0138] The data in the bottleneck layer 313 is data (15 x 48 channels) that comprehensively represents the characteristics of the input data 311D.

[0139] Furthermore, the data from the bottleneck layer 313 is subjected to upsampling, normalization, and deconvolution processes in each layer of the decoder 314, and converted into output data 315D (one-dimensional vector data) with 240 data points, which is then output by the output layer 315.

[0140] The output data 315D output from the output layer 315 is output as data showing the heart rate cycle waveform.

[0141] During the learning phase, a loss function F1 is calculated based on the error (mean squared error or mean absolute error, etc.) between the output data 315D and the ground truth data 318 of the heart rate cycle waveform. The learning model 410 (specifically its parameters, etc.) is then trained to minimize (optimize) this loss function F1. In other words, the learning model 410 is trained so that the output data 315D (heart rate cycle waveform) approaches the ground truth data 318 (ground truth waveform), and a trained model 420 is generated.

[0142] Figure 15 shows an example of input and output to a learning model. As shown in Figure 15, when N data sequences (L21, L21, L21,...) are input to the learning model 410 (particularly the trained model 420), the learning model 410 outputs a heart rate cycle waveform (curve L51) as output data 315D.

[0143] <1-5. Processing during the learning phase> The following describes the processing of the learning phase of the learning model 410 and the inference phase using the learning model 410 (trained model 420) in order.

[0144] First, let's explain the process during the learning phase.

[0145] Figure 11 is a flowchart illustrating the processing during the learning phase. The processing shown in Figure 11 is executed by the controller 31, etc. Figure 11 also illustrates the method for generating a learning model (trained model).

[0146] The learning process shown in Figure 11 can be broadly divided into two parts: the preparation of training data (steps S11 to S15) and the machine learning process of the learning model 410 using the training data (step S16).

[0147] In step S11, the heart rate measurement device 10 acquires N data sequences of the person to be used for training data acquisition by measurement using the millimeter wave sensor 20 (measurement of reflected waves from the person). For example, N data sequences Φn(τ) (n=1,...,N) are acquired for multiple combinations Cn(r,θ) of the person.

[0148] Simultaneously, the heart rate waveform of the person is measured and acquired using a PPG sensor (step S12). Steps S11 and S12 are executed simultaneously and in parallel.

[0149] In step S13, training data is generated, comprising N data sequences (of the person) obtained in step S11 and the correct data of the heart rate cycle waveform (of the person) obtained in step S12.

[0150] The processes described in steps S11 to S13 are repeatedly performed for multiple subjects to generate multiple training data sets. Once the process of generating training data for a predetermined number of people is complete, the process proceeds from step S14 through step S15 to step S16. Step S15 (processing to increase training data (data augmentation process), etc.) will be explained later.

[0151] In step S16, the heart rate measurement device 10 (also referred to as the learning model generation device) performs machine learning on the learning model 410 using the multiple training data generated in steps S11 to S15. Specifically, machine learning is performed to reduce the error (more precisely, the cumulative value of the error for each frame) between the heart rate cycle waveform output from the learning model 410 and the heart rate cycle waveform of the correct data (in other words, to minimize the loss function related to the said error).

[0152] This machine learning process generates a trained model 420 (a trained model 410).

[0153] According to the trained model 420, as shown in Figure 15, the heart rate cycle waveform (curve L51) of the person being measured is estimated based on N data sequences at N locations on the person being measured (data sequences showing the time change Φn(τ) of the phase Φ of N complex signals (n=1,...,N), etc.).

[0154] <1-6. Details of the learning phase processing> Next, we will explain the details of the learning phase processing.

[0155] <Dropout> During the learning phase, a dropout process is performed. For example, a process is performed (dropout process) in which the features of a portion of the bottleneck layer 313 (a specific percentage (e.g., 50%)) are overwritten with zeros.

[0156] Dropout processing makes it possible to suppress dependence on specific features. Even if a data sequence with a specific feature is not included in multiple data sequences, a learning model 410 that can accurately estimate the heart rate cycle waveform can be generated based on other data sequences with other features (within those multiple data sequences).

[0157] <Time Scale Conversion> Furthermore, it is preferable that, in addition to the training data acquired in steps S11 to S14, data generated (enlarged) based on said training data (in step S15) is also generated as new training data (additional training data).

[0158] For example, in step S15, data augmentation processing is performed, which involves the following time scale transformation (see Figure 17). Figure 17 is a diagram illustrating the time scale transformation.

[0159] In detail, not only is training data composed of multiple data sequences and ground truth data of heart rate cycle waveforms generated, but new training data is also generated, which is composed of modified ground truth data obtained by modifying the training data. In detail, the training data is composed of modified multiple data sequences in which the time scale of the multiple data sequences has been changed (stretched or compressed) by a predetermined percentage (e.g., 20% reduction, 10% reduction, 10% increase, and 20% increase), and data in which the time scale of the ground truth data has been changed by the same predetermined percentage.

[0160] For example, by using only 400 of the initial 500 sampled data points (the first 4 seconds of data) (see the middle section of Figure 17) as the input data for the first 5 seconds (see the top section of Figure 17), a data sequence with a 20% reduced time scale is generated. In short, a slowed-down data sequence is generated. Such data sequences are obtained for all or some of the N combinations Cn, and multiple data sequences are generated with the time scale changed by a predetermined percentage (-20%). Similarly, by treating only 400 of the 500 sampled data points of the ground truth heart rate cycle waveform (PPG waveform, etc.) as the 5-second ground truth data, ground truth data with a 20% reduced time scale is generated. Then, new training data is generated, consisting of these multiple data sequences and the new ground truth data. This new training data corresponds to training data with a heart rate reduced by 20% compared to the original training data. For example, if the heart rate of the original training data is 60, the heart rate of the new training data is 48.

[0161] Furthermore, for example, by using 600 sampled data points (data from the first 6 seconds) (see bottom of Figure 17) as the input data for the first 5 seconds, including not only the first 500 sampled data points but also the following 100 (see bottom of Figure 17), a data sequence with a 20% increased time scale is generated. In short, a sped-up data sequence is generated. Such data sequences are obtained for all or some of the N combinations Cn, and multiple data sequences are generated with the time scale changed by a predetermined percentage (+20%). Similarly, by treating 600 sampled data points of the correct heart rate cycle waveform (PPG waveform, etc.) as 5 seconds of correct data, correct data with a 20% increased time scale is generated. Then, new training data is generated, consisting of these multiple data sequences and the new correct data. This new training data corresponds to training data with a heart rate increased by 20% compared to the original training data. For example, if the heart rate of the original training data is 60, the heart rate of the new training data is 72.

[0162] Then, in step S16, the learning model 410 is trained not only on the original training data but also on these new training data.

[0163] The data generation process described above makes it possible to create various heart rate data. By using diverse data for training, it is possible to improve the accuracy of the training model 410.

[0164] <Channel Shuffling> In step S15, data augmentation processing involving channel shuffling (see Figure 18) may also be performed. Figure 18 is a diagram illustrating channel shuffling.

[0165] As described above, the input data 311D has multiple channels (a multi-channel structure). Specifically, the input data 311D (see Figure 13) has a structure in which data sequences (one-dimensional array data consisting of 240 elements) for each complex signal of 192 spatial positions Cn(r, θ) are arranged in individual channels on a data sequence basis (a total of 192 channels) (see upper part of Figure 18).

[0166] As such input data 311D, data is further generated by changing (shuffling) the channel arrangement of multiple data sequences. In other words, input data is further generated by changing which of the multiple channels each data sequence is assigned to (placed in).

[0167] The upper part of Figure 18 shows input data (first input data) in which multiple data sequences are arranged for multiple channels in a first configuration. On the other hand, the lower part of Figure 18 shows input data (second input data) in which multiple data sequences are arranged for multiple channels in a second configuration (a different configuration from the first configuration). Specifically, the third data sequence from the left and the fourth data sequence from the right (189th from the left) in the first input data in the upper part of Figure 18 are swapped to generate the second input data in the lower part of Figure 18. In this way, new input data is generated in which the channel positions within the original input data 311D have been changed. Then, new training data is generated consisting of the second input data (after the channel position change) and the correct answer data (the same correct answer data) corresponding to the original first input data. In other words, multiple input data for the same correct answer data are generated, and multiple training data sets are generated. Furthermore, in step S16, the learning process is performed based on this new training data as well.

[0168] By using these multiple training data sets, it is possible to train the learning model 410 independently of the channel configuration of each data sequence in the training data (which channel each data sequence is placed in). For example, the learning model 410 can output the same (target) heart rate cycle waveform (the same waveform) for either of two input data sets that differ only in their channel configuration.

[0169] It is preferable that a large number of new input data, including channel shuffling, and a large number of training data using such input data are generated, and that the learning model 410 is trained based on the large number of training data.

[0170] Note that while this example shows two data columns swapped, it is not limited to this; three or more data columns may be swapped.

[0171] <Channel Drop> In step S15, data augmentation processing involving channel dropping (see Figure 19) may also be performed. Figure 19 is a diagram illustrating channel dropping.

[0172] Specifically, data obtained by replacing some of the data sequences (data sequences from some channels in a multi-channel dataset) of multiple data sequences relating to multiple combinations Cn with zeros may also be used as training data, and the learning model 410 may be trained using this data.

[0173] For example, as shown in Figure 19, new input data is generated by replacing all elements of a predetermined percentage (for example, 10%) of the original input data 311D (for example, 19 elements, which is 10% of 192 elements (only the 5th from the left and the 3rd from the right are shown in Figure 19)) of the data sequence with zeros. In other words, new input data is generated by disabling some channels of a multi-data sequence. Then, new training data is generated, consisting of this new input data and the correct answer data 318 corresponding to the original input data 311D. Furthermore, in step S16, the learning model 410 is trained based on this new training data as well.

[0174] Such channel dropping allows the learning model 410 to be trained to estimate heart rate waveforms based on various data sequences, without depending on a specific data sequence. For example, the learning model 410 can be trained to estimate heart rate waveforms from data sequences located far from the heart, without using data sequences located close to the heart.

[0175] <1-7. Inference Stage Processing> Next, we will explain the inference stage processing using the trained model 420 (the process of estimating the heart rate cycle waveform for a new person).

[0176] Figure 12 is a flowchart showing the processing at the inference stage. The processing shown in Figure 12 is performed by the controller 31, etc. Through this inference stage processing, the heart rate cycle waveform of the person to be estimated is estimated based on multiple data sequences related to the person to be estimated.

[0177] First, in step S31, N data sequences (for example, a data sequence representing the time change of phase (Φn(τ) (n=1,...,N))) are obtained for multiple combinations (spatial positions) Cn(r,θ) of the estimated target person by measurement using the millimeter-wave sensor 20.

[0178] In the next step S32, when N data sequences relating to the person to be estimated are input to the trained model 420, the heart rate cycle waveform of the person to be estimated (see curve L51 in Figures 15 and 16, etc.) is calculated by the trained model 420 as the estimation result (estimated waveform) and output from the trained model 420.

[0179] In step S33, the heart rate and other values ​​are calculated based on the inference results (heart rate cycle waveform) from the trained model 420 (trained learning model 410). Specifically, the heart rate measurement device 10 performs peak detection processing on the estimated waveform (heart rate cycle waveform) output from the learning model 410 and detects multiple peaks. Each peak can be detected using various methods such as differential calculus (a method that determines a peak as a point where the first derivative of the data is zero) or thresholding (a method that determines a peak when it exceeds a threshold). Furthermore, each peak may be a peak of a local maximum, but is not limited to this, and may also be a peak of a local minimum.

[0180] Then, the interval between each peak and the peaks adjacent to it (adjacent peaks) (peak interval) is calculated for multiple peaks, and multiple peak intervals are obtained (see Figure 16).

[0181] Furthermore, a representative value (representative peak interval) is determined based on multiple peak intervals. Examples of representative values ​​include the median (or mode or mean, etc.) of the multiple peak intervals. Then, the person's heart rate is calculated based on this representative peak interval. For example, the person's heart rate is calculated by dividing the median of the multiple peak intervals by "60". In detail, if multiple (for example, five) peak intervals are "0.72", "0.76", "0.80", "0.81", and "0.82" (seconds), the value "75" (= 60 / 0.8) (BPM: Beats Per Minute) obtained by dividing the median of these peak intervals "0.80" by 60 is calculated as the person's heart rate (see Figure 16). Figure 16 shows a curve L21 for one of the N data sequences related to the subject being estimated, and a heart rate cycle waveform (curve L51) estimated by the learning model 410 based on that N data sequence.

[0182] <1-8. Effects of the Embodiment> In the above embodiment, the learning model 410 estimates the heart rate cycle waveform of a single person being measured based on N signals (N data sequences) at N spatial locations. This makes it possible to estimate the heart rate cycle waveform more accurately compared to estimating the heart rate cycle waveform based on a signal at a single location.

[0183] In particular, if the heart rate cycle waveform is estimated based on a signal from a single location, the accuracy of the heart rate cycle waveform may change depending on the validity of that single location. On the other hand, as in the embodiment described above, when the heart rate cycle waveform is estimated based on multiple signals from multiple (N) spatial locations, it is possible to suppress such changes in accuracy (variability) and estimate the heart rate cycle waveform more accurately.

[0184] Furthermore, according to the above embodiment, the heart rate cycle waveform is estimated using N data sequences relating to N spatial positions. Therefore, it is possible to obtain a more accurate heart rate cycle waveform (estimated waveform) without having to pre-specify (select) a single position (one location) among the N spatial positions.

[0185] Furthermore, based on the heart rate cycle waveform estimated by the learning model 410, it is possible to obtain (calculate) the heart rate interval (peak interval) for each beat. Specifically, multiple peak positions (peak times) are detected based on the heart rate cycle waveform, and multiple peak intervals are determined based on these multiple peak positions. By detecting whether or not there is a regular change in the heart rate interval for each beat, it is also possible to evaluate whether or not a person is under stress. For example, if the heart rate interval is changing very regularly, it can be determined that the person is under stress.

[0186] Furthermore, it is possible to estimate (measure) the heart rate based on the heart cycle waveform. Specifically, the heart rate can be calculated by determining the median (or average value, etc.) of the multiple peak intervals based on the heart cycle waveform.

[0187] In particular, according to the above embodiment, the estimation process using the learning model 410 yields a heart cycle waveform (estimated heart cycle waveform) that is close to the true heart cycle waveform. Specifically, even if there are subtle differences in waveform shape between the estimated heart cycle waveform and the true heart cycle waveform, each peak interval of the estimated heart cycle waveform can be estimated to be very close to each peak interval of the true heart cycle waveform. Therefore, the representative peak interval (heart rate) can also be estimated well as a value close to the true value.

[0188] <1-9. Modification 1 of the First Embodiment: Reliability> The hourglass-shaped network model 310 (see Figure 13) in the above embodiment is a model for estimating the heart rate cycle waveform. However, it is not limited to this, and an hourglass-shaped network model 330 (see Figure 20) that estimates not only the heart rate cycle waveform but also the reliability of the heart rate cycle waveform may be used.

[0189] The hourglass-shaped network model 330 in Figure 20 has a structure in which a subnetwork 320 having a feature extraction layer is added to the above-described hourglass-shaped network model (also simply called an hourglass-shaped network) 310.

[0190] Subnetwork 320 takes the final layer (size: 240 x 48) of the decoder 314 of the hourglass-shaped network 310 as input and has a structure that further includes multiple convolutional layers (encoders), etc. Subnetwork 320 ultimately outputs a single value (scalar) as output data 325.

[0191] This subnetwork 320 is machine-trained, for example, after the hourglass-shaped network model 310 has finished learning, with the parameters of the hourglass-shaped network model 310 already applied.

[0192] During training, the error in the peak interval (rri: RR-Interval) of the heart rate cycle waveform (the error between the estimated value of the representative peak interval and the true value of the representative peak interval) is assigned as ground truth data 328. The ground truth data 328 for the error in the (representative) peak interval of the heart rate cycle waveform is the error between the following two peak intervals P1 and P2. Peak interval P1 (estimated value of the peak interval) is the representative peak interval (mean or median, etc.) calculated based on the output data 315D (estimated heart rate cycle waveform) of the hourglass-shaped network model 310. On the other hand, peak interval P2 (true value of the peak interval) is the representative peak interval (mean or median, etc.) calculated based on the ground truth data 318 of the heart rate cycle waveform (PPG waveform, etc.). Then, the subnetwork 320 is trained to minimize the loss function F2 (the absolute value of the difference, etc.) related to the difference between the output data 325 and the ground truth data 328, using training data where the error between the two peak intervals P1 and P2 is the ground truth data 328.

[0193] As a result, the network model 320 is trained to output the above scalar (output data 325) as an estimated value of the error in the peak interval (the error between the two peak intervals P1 and P2) based on the heart rate cycle waveform estimated by the hourglass-shaped network model 310. In other words, even when the correct data 318 of the heart rate cycle waveform is unknown, it is possible to obtain an estimated value of the error (the error between the estimated value P1 of the representative peak interval and the true value P2 of the representative peak interval).

[0194] During inference, the heart rate cycle waveform (output data 315D) is estimated, and the estimation error (output data 325) of the representative peak interval (heart rate) of the heart rate cycle waveform is calculated.

[0195] Furthermore, the estimated value of this error (output data 325) can also be expressed as a value representing the reliability of the heart rate cycle waveform by the hourglass-shaped network model 310 (more specifically, the reliability of the representative peak interval (heart rate) of the heart rate cycle waveform).

[0196] For example, when the correct data 318 is normalized so that its values ​​are between 0 and 1, the confidence level is expressed as the estimated error value (estimated error value) minus 1. More specifically, if the estimated error value is 0.50, the confidence level is 50% (= 1 - 0.5); if the estimated error value is 0.10, the confidence level is 90% (= 1 - 0.1); and if the estimated error value is 0.01, the confidence level is 99% (= 1 - 0.01).

[0197] Note that the subnetwork 320 may, but is not limited to, being machine-trained with the parameters of the trained hourglass network 310 already applied. For example, the hourglass network 310 and the subnetwork 320 may be machine-trained simultaneously. More specifically, the learning model 410 may be machine-trained to minimize a new loss function F3, which is expressed as a linear sum (e.g., F1 + F2 × α1) of a loss function F1 relating to the difference between the output data 315D and the ground truth data 318 of the heart rate cycle waveform, and a loss function F2 relating to the difference between the output data 325 and the ground truth data 328. Here, the value α1 is a parameter that adjusts the balance of learning in order to correctly learn both the heart rate cycle waveform and the confidence level.

[0198] <Further variations regarding reliability> Here, a scalar representing the overall reliability of the heart rate cycle waveform (output data 315D) is calculated as output data 325. In other words, an example is given where both output data 325 and the correct data 328 are scalars.

[0199] However, this is not limited to this. For example, a vector showing the confidence level at each point in time of the heart rate cycle waveform (output data 315D) may be output as output data 325 (also referred to as 325b). In other words, both output data 325 and ground truth data 328 may be vectors.

[0200] In detail, for example, by upsampling from the downsampled intermediate layer (such as the final layer of the subnetwork 320) in the subnetwork 320, a vector of size 240 × 1 is output as output data 325 (325b (not shown)).

[0201] During training, a vector composed of the difference values ​​at each time point between the heart rate cycle waveform (estimated waveform) and the PPG waveform, etc. (true waveform) is used as the ground truth data 328. Specifically, each element of the ground truth data (vector) 328 is the difference value between the corresponding elements of the two vectors: output data 315D (a vector whose elements are the values ​​of the estimated heart rate cycle waveform at each time point) and ground truth data 318 (a vector whose elements are the values ​​of the true heart rate cycle waveform (PPG waveform, etc.) at each time point). In short, a vector showing the difference (error) between the two waveforms at each time point is calculated as the ground truth data 328. The subnetwork 320 is then trained so that the output data 325b approaches the ground truth data 328. More specifically, the subnetwork 320 is trained to minimize the loss function F2 (mean squared error or mean absolute error, etc.) related to the difference between the output data (vector) 325b and the ground truth data (vector) 328.

[0202] During inference using the trained network 330 (trained model), the heart rate cycle waveform (output data 315D) is estimated, and the time-dependent estimation error of the heart rate cycle waveform (the estimation error for each time point of the heart rate cycle waveform) is calculated as output data 325. In other words, the subnetwork 320 is trained so that the output data 325b (vector) constitutes a collection of estimated error values ​​for each time point related to the heart rate cycle waveform. Furthermore, each element of the output data 325 (vector) (the estimated error value for each time point) also represents a value that indicates the confidence level of the heart rate cycle waveform at each time point. In other words, the confidence level of the heart rate cycle waveform (more specifically, the confidence level for each time point of the heart rate cycle waveform) is also calculated. The conversion process from error to confidence level can be performed in the same way as in the case of a scalar as described above.

[0203] <1-10. Modification 2 of the First Embodiment: Diffusion Model> In the above embodiment, an hourglass-shaped network model 310 is exemplified as the learning model 410. However, it is not limited to this, and a diffusion model 350 (see Figure 21) may also be adopted as the learning model 410. The diffusion model 350 is also one of the neural network models (deep neural network models).

[0204] Figure 21 shows an example of the diffusion model 350.

[0205] The diffusion model 350 has a diffusion process that converts the heart rate cycle waveform to noise (gradually in multiple steps) and a despreading process that converts the noise back to the heart rate cycle waveform (gradually in multiple steps). The diffusion model 350 is a learning model that learns learning parameters (parameters for removing noise and restoring the heart rate cycle waveform) in the despreading process. U-Net and the like are used as part of the diffusion model 350.

[0206] The diffusion model 350 is a conditional diffusion model. More specifically, the diffusion model 350 is a learning model that takes a condition vector 353 and noise 354 as inputs and outputs a heart rate cycle waveform (output data 356D).

[0207] Here, as a condition, a condition vector 353 is assigned to associate the estimated output waveform (output data 356D) from the diffusion model 350 with the specific waveform from which the estimation was performed. In other words, a condition (condition vector 353) is assigned to cause the estimated output waveform corresponding to the specific waveform from which the estimation was performed to be output as output data 356D.

[0208] Specifically, the waveform data (vectorized) of the heart rate cycle waveform generated (estimated) by the encoder 352 based on the N data sequences to be learned is used as the condition vector 353. In other words, this waveform data (estimated waveform by encoder 352) is used as the specific waveform (one of the input data) that the diffusion model 350 estimates.

[0209] The diffusion model 350 is machine-learned to generate (output) a waveform corresponding to the specific waveform of the source of estimation as the estimated heart rate cycle waveform (output data 356D) based on the conditions. While diffusion models are generally used in text-to-image image generation models (image generation models that generate images according to text), this diffusion model 350 is, so to speak, a waveform (specific waveform) to waveform (estimated heart rate cycle waveform) generation model.

[0210] The condition vector 353 is generated as follows.

[0211] Specifically, N data sequences relating to N complex signals for N combinations (spatial positions) Cn(r, θ) are input to the input layer 351. The N data sequences are, for example, the time variation Φn(τ) of the phase Φ of the complex signal (n=1,...,N). If the size of one data sequence is 240 × 1, then in the case of N = 192, input data 311D (Figure 13) of size 240 × 192 is used as input data 351D (Figure 21) to the input layer 351.

[0212] The input data 351D is then converted into a condition vector 353 (for example, size: 240 × 1) via the encoder 352. For example, the hourglass-shaped network model 310 described above can be used as the encoder 352. Note that the encoder 352 is not limited to the hourglass-shaped network model 310; it must be capable of converting the time-dependent changes of signals at multiple locations into a one-dimensional vector (such as the time-dependent changes of a single signal).

[0213] The diffusion model 350 is trained using the correct data 358 (PPG waveform, etc.) with the condition (condition vector 353) applied, so that the diffusion model 350 (learned model 410) is trained to output a heart rate cycle waveform corresponding to the condition (specific waveform) as output data 356D.

[0214] More specifically, when input data (a sequence of N data points) 351D, which is the source of the condition vector, is input, the encoder 352 generates a condition vector 353, and this condition vector 353 is input to the diffusion model 350 (along with noise 354). The diffusion model 350 is trained to output a heart rate cycle waveform (output data 356D) corresponding to the condition vector 353 (the heart rate cycle waveform estimated by the encoder 352 based on the N data points) as a waveform that is close to the ground truth data 358.

[0215] Subsequently, N data sequences relating to the person to be estimated are input as input data 351D and converted into a condition vector 353 by the encoder 352. This condition vector 353 is then input (along with noise 354) to the trained diffusion model 350 (trained model 420). Accordingly, the heart rate cycle waveform relating to the person to be estimated is obtained as output data 356D from the diffusion model 350. In other words, it is possible to estimate the heart rate cycle waveform relating to the person to be estimated.

[0216] Such a diffusion model 350 may be used to estimate the heart rate cycle waveform. This makes it possible to obtain the same effects as in the first embodiment described above.

[0217] Furthermore, by using the diffusion model 350, subtle differences in each measurement of PPG waveforms (ground truth data) can be absorbed by differences in input noise data (noise vectors), thus facilitating better learning.

[0218] Furthermore, time scale transformation, channel dropping, and channel shuffling may also be performed during the training of the diffusion model 350 (and 370 (described below)).

[0219] <1-11. Modification 3 of the First Embodiment: Diffusion Model (with Confidence Level)> The diffusion model 350 may also be a model 370 (see Figure 22) that calculates confidence levels. The diffusion model 370 in Figure 22 has a structure in which a subnetwork 320b having a feature extraction layer is added to the diffusion model 350 described above. The subnetwork 320b has the same structure as the subnetwork 320 (see Figure 20). However, the subnetwork 320b has a structure that takes the final layer (size: 240 × 48) of the encoder in the diffusion model 350 as input and further includes a plurality of convolutional layers (encoders), etc.

[0220] Using such a diffusion model 370, output data 356D (estimated heart rate cycle waveform) may be output, along with the confidence level (estimation error) for the said estimated heart rate cycle waveform. Furthermore, the output data 356D may be a scalar or a vector.

[0221] <1-12. Other Modifications of the First Embodiment, etc.> In the above embodiment, etc., a predetermined fixed area is used as the predetermined area (measurement target area) 110 (see Figures 6 and 9), but the embodiment is not limited thereto. The measurement target area may vary depending on the signal strength measured by the millimeter-wave sensor 20. In particular, when the measurement position of a person may change due to the movement of the person, it is preferable that the measurement target area (area for use of IQ data) is determined according to the position of the person after the movement.

[0222] For example, the measurement target area may be determined based on a location within the measurable space that has a signal strength of a predetermined level or higher. More specifically, a rectangular area (or parallelogram-shaped area, etc.) of a predetermined size centered on a location within the measurable space that has a signal strength of a predetermined level or higher (e.g., the location with the highest signal strength) may be determined as the measurement target area. Alternatively, a predetermined number (for example, 192) of segmented areas (segmented spaces) (small areas corresponding to radar cubes) that include a location with a signal strength of a predetermined level or higher may be selected from the maximum measurable range (measurable space) of the millimeter-wave sensor 20, and the collection of the selected segmented areas may be determined as the measurement target area. In this way, the multiple spatial locations to be used may be determined to include locations with a signal strength of a predetermined level or higher.

[0223] Alternatively, a set of a predetermined number (for example, 192) of segmented regions selected from the maximum measurable range in order of decreasing signal intensity may be determined as the measurement target region.

[0224] The signal strength can be, for example, the norm of the IQ data after the effects of static clutter have been removed (more specifically, the maximum or average value of the norm within a certain period). However, it is not limited to this, and the norm of the IQ data before the effects of static clutter have been removed may also be used. Furthermore, if static clutter removal is not performed, the "change" in the signal will be used (calculated), so the virtual origin (see Figure 7) can be placed anywhere (it may be placed in a position other than the actual origin).

[0225] Thus, the multiple spatial locations to be used may be determined based on the signal intensity of multiple spatial locations within the maximum measurable range (measurable space).

[0226] Furthermore, in the above embodiments, a data sequence relating to a complex signal over a 5-second period (such as the time change Φn(τ) of the phase of the complex signal) is used, but the invention is not limited to this. A data sequence relating to a complex signal of a different period length (for example, 30 seconds) may be used as input data 311D for the learning model 410. However, by using a shorter period length as input data, it is possible to learn the learning model 410 well relatively easily (with a relatively small amount of learning).

[0227] Furthermore, in the above embodiments, a data sequence relating to the complex signal for the first 5 seconds (such as the time change Φn(τ) of the phase of the complex signal) is used, but the method is not limited to this. For example, a data sequence relating to the complex signal for 5 seconds starting from a predetermined point in time other than the beginning (for example, 10 seconds after the start of measurement) may be used. Alternatively, data for each 5-second interval obtained by dividing a 30-second period (data for each of the 6 intervals in total) may be adopted as input data 311D, and a heart rate cycle waveform may be estimated based on each of the data for the 6 intervals. Furthermore, based on the 6 heart rate cycle waveforms estimated based on the data for the 6 intervals, a peak interval and a representative value of the peak interval (representative peak interval) may be determined. For example, if approximately five peaks and peak intervals (about four to ten) are detected from each five-second heart cycle waveform, and a total of approximately 30 peak intervals (about 24 to 60) are detected from six heart cycle waveforms, the representative value (mean or median, etc.) of these approximately 30 peak intervals may be calculated as the person's heart rate.

[0228] Alternatively, as described below, the learning model 410 may estimate the heart rate cycle waveform for each of several different periods (each having a predetermined duration (e.g., 5 seconds)) by sliding the start time of the period by a predetermined minute amount of time Δτ0 (for example, a minute amount of time corresponding to one to several frames). Furthermore, based on the multiple heart rate cycle waveforms estimated for these multiple periods, the peak interval and a representative value of the peak interval (representative peak interval) may be determined.

[0229] For example, the learning model 410 may estimate 10 different heart rate cycle waveforms for 10 different periods (each having a predetermined duration, e.g., 5 seconds) whose start times are shifted by 10 ms (milliseconds). Alternatively, a representative value (mean or median, etc.) of the peak interval obtained from the 10 heart rate cycle waveforms may be calculated as the person's heart rate.

[0230] <2. Second Embodiment> The second embodiment is another modification of the first embodiment. The following will focus on the differences from the first embodiment and the like.

[0231] In the first embodiment described above, a data sequence relating to the complex signal for only the first five seconds (such as the time change Φn(τ) of the phase of the complex signal) is used.

[0232] On the other hand, in this second embodiment, not only the first five seconds but also the data sequence relating to the complex signal for periods other than the first five seconds (such as the time change Φn(τ) of the phase of the complex signal) is used.

[0233] Specifically, during the inference phase, for each of the multiple distinct periods Bi (i=1, ..., M) (see Figure 23), each having a predetermined period length, with the period start time shifted by a predetermined small time Δτ0, multiple data sequences (curves L21) spanning the predetermined period length are input to the learning model 410 (420). Then, the heart rate cycle waveform (curve L51) output from the learning model 410 in response to this input is acquired as a provisional output waveform (provisional output waveform). As a result, multiple heart rate cycle waveforms (multiple provisional output waveforms) for multiple periods are acquired. Then, by combining the multiple provisional output waveforms acquired for the multiple periods (after aligning the time), a heart rate cycle waveform (integrated heart rate cycle waveform) for the person is generated.

[0234] In detail, as shown in Figure 23, first, for each of M (for example, 10) distinct periods B1 to B10 (each having a predetermined period length (for example, 5 seconds)) whose period start time is shifted by a predetermined small time Δτ0 (for example, 10 ms (milliseconds)) (one frame), N (for example, 192) data sequences (one set of data) are obtained over the predetermined period length. Then, in response to the input of one set of data (N data sequences) for each period Bi into the learning model 410 (420), a provisional output waveform (curve L51) for each period Bi is output. By performing this process for M periods Bi (for example, B1 to B10), M heart rate cycle waveforms (M provisional output waveforms) for the M periods are obtained (see the leftmost column of Figure 23).

[0235] Furthermore, as shown in Figure 24, the M provisional output waveforms acquired for the M periods are combined (after aligning the time). This generates a heart cycle waveform for the person being measured (an integrated heart cycle waveform, also called a combined heart cycle waveform or synthesized heart cycle waveform). Specifically, the average value of the time τ values ​​of the M provisional output waveforms (average value for each time τ) should be obtained as the signal intensity of the integrated heart cycle waveform at each time τ. In this way, the integrated heart cycle waveform is generated.

[0236] The M provisional output waveforms are obtained as waveforms that are roughly similar to each other but slightly different from each other. By calculating the average waveform of these M provisional output waveforms, it is possible to obtain a more accurate heart cycle waveform (integrated heart cycle waveform). For example, as shown in Figure 24, the integrated heart cycle waveform obtained by averaging the M provisional output waveforms does not contain the false peaks (peaks that are not actually peaks) (shown by thick arrows in Figure 24) that were present only in some of the M provisional output waveforms.

[0237] Furthermore, based on this integrated heart cycle waveform, it is possible to obtain (calculate) the heart rate interval for each individual beat. Specifically, multiple peak positions are detected based on the integrated heart cycle waveform, and multiple peak intervals are determined based on these multiple peak positions.

[0238] Furthermore, it is possible to estimate (measure) the heart rate based on the integrated cardiac cycle waveform. Specifically, it is possible to calculate the heart rate based on the multiple peak intervals derived from the integrated cardiac cycle waveform.

[0239] <Modification of the Second Embodiment> In the second embodiment described above, an integrated heart cycle waveform is generated using all the values ​​of the M provisional output waveforms, but the embodiment is not limited to this. For example, if the learning model 410 also outputs a confidence score for the output data 315D (see Figures 20 and 22), the integrated heart cycle waveform may also be generated based on the confidence score of each provisional output waveform.

[0240] In detail, for example, if one confidence level (scalar) is output for one estimated heart rate cycle waveform (provisional output waveform) (when output data 325 (see Figures 20 and 22) is a scalar), then among the M provisional output waveforms, those provisional output waveforms with a confidence level below a predetermined threshold may not be used. In detail, among the 10 provisional output waveforms, two provisional output waveforms with a confidence level below a predetermined threshold may not be used, and the remaining eight provisional output waveforms may be used to generate the integrated heart rate cycle waveform. Alternatively, the values ​​of the M provisional output waveforms at each time point may be weighted and averaged according to the respective confidence levels of the M provisional output waveforms.

[0241] Furthermore, if a confidence level is output for each estimated heart rate cycle waveform (provisional output waveform) at each point in time (when the output data 325 (see Figures 20 and 22) is a vector), then among the values ​​of the M provisional output waveforms at each point in time, the values ​​at points in time where the confidence level is below a predetermined threshold (for example, 50%) do not need to be used. Specifically, among the values ​​of 10 provisional output waveforms (10 values) at a certain point in time, 2 values ​​with a confidence level below a predetermined threshold may not be used, and the remaining 8 values ​​may be used to calculate the value at that point in time (the integrated heart rate cycle waveform value). In this way, the value at each point in time may be calculated based on the confidence level at each point in time, and the integrated heart rate cycle waveform may be obtained. Alternatively, the values ​​at each point in time of the M provisional output waveforms may be weighted and averaged according to the confidence level at each point in time of the M provisional output waveforms to generate the integrated heart rate cycle waveform.

[0242] <3. Third Embodiment> The third embodiment is a modification of the second embodiment. The following will focus on explaining the differences from the second embodiment and the like.

[0243] In the second embodiment described above, an example is shown in which M provisional output waveforms obtained for M different periods are combined to generate an integrated heart cycle waveform (see Figure 24).

[0244] In this third embodiment, an example is given in which a peak position (integrated peak position) based on M provisional output waveforms obtained for M different periods is determined without determining an integrated heart cycle waveform (see Figure 25). The peak position based on the M provisional output waveforms is a peak position (peak time) obtained by integrating (combining) the peak positions of the M provisional output waveforms, and is also called the integrated peak position (or combined peak position).

[0245] Specifically, in the inference stage, similar to the second embodiment, first, M (for example, 10) heart rate cycle waveforms (provisional output waveforms) (curves L51) are generated for M different periods. Then, in this embodiment, without synthesizing the M heart rate cycle waveforms, a peak position detection process is performed based on each of the M individual heart rate cycle waveforms (see Figure 25). As a result, approximately 5 (4 to 10) peak positions are obtained for each of the M heart rate cycle waveforms.

[0246] Then, based on the peak positions of each provisional output waveform acquired for each of the M periods Bi, peak synthesis is performed to determine the final peak position of the biological heart cycle waveform (synthesized (integrated) peak position (integrated peak position)) (see Figure 25).

[0247] In this scenario, when M heart rate cycle waveforms are arranged to synchronize their timing, the peak positions of the M provisional output waveforms should ideally be at the same position (time). However, although the M heart rate cycle waveforms are roughly similar to each other, they are slightly different from one another. Therefore, the peak positions of the M heart rate cycle waveforms may be shifted.

[0248] In this embodiment, the number of peaks whose peak positions (peak times) match among the multiple peaks in the M provisional output waveforms is counted (for each peak position). In other words, for the M provisional output waveforms, the cumulative number of peaks (additional count) at each position (time) is counted. This cumulative count is the total number of peaks detected at a certain time (the same time) among the peaks of the M provisional output waveforms. For example, for each time T1 to T7 in Figure 25, the cumulative counts are 9, 10, 10, 1, 10, 9, and 9, respectively. In particular, at time T4, a peak is detected only from the provisional output waveform of period B2, and the cumulative count is only 1.

[0249] Thus, at most of the peak positions (peak times) detected from each of the M provisional output waveforms (six peaks other than time T4 in Figure 25), there are nearly M peaks (for example, 10 to 7 peaks) (the peak positions near M match). On the other hand, at a certain peak position, there may be fewer than a predetermined number of peaks (for example, only one peak) (only peaks less than or equal to the predetermined number of peaks out of M match). Such peak positions are so-called false positives (see the thick arrows in Figure 25, etc.).

[0250] Peak locations corresponding to only a predetermined number of peaks or fewer are excluded, and the integrated peak location is determined. Specifically, if a predetermined number or more peaks exist at a certain location (the same location), it is determined that a peak exists at that location (i.e., that location is a peak location (integrated peak location)). On the other hand, if fewer than a predetermined number of peaks exist at a certain location (the same location), it is determined that no peak exists at that location (i.e., that location is not a peak location (integrated peak location)). In Figure 25, the peak at time T4 is excluded (as a false positive), and the peak locations at times T1-T3 and T5-T7 are detected as the integrated peak location.

[0251] In this way, the peak position (integrated peak position) is determined based on M provisional output waveforms.

[0252] Furthermore, this integrated peak position makes it possible to determine the interval between each heartbeat. It also makes it possible to estimate (measure) the heart rate based on the integrated peak position. Specifically, it is possible to calculate the heart rate based on multiple peak intervals derived from the integrated peak position.

[0253] <Modification of the Third Embodiment> In the third embodiment described above, the integrated peak position (integrated peak time) is determined based on all the peak positions detected in each of the M provisional output waveforms, but the embodiment is not limited to this. For example, if the learning model 410 also outputs a confidence score for the output data 315D (see Figures 20 and 22, etc.), the integrated peak position may also be determined based on the confidence score of each provisional output waveform.

[0254] In detail, for example, if one confidence level (scalar) is output for one estimated heart rate cycle waveform (provisional output waveform) (when output data 325 (see Figures 20 and 22) is a scalar), then among the M provisional output waveforms, those provisional output waveforms with a confidence level below a predetermined threshold do not need to be used. For example, among 10 provisional output waveforms, 2 provisional output waveforms with a confidence level below a predetermined threshold may not be used, and the remaining 8 provisional output waveforms (and their peak positions) may be used to determine the integrated peak position.

[0255] Furthermore, if a confidence level is output for each estimated heart rate cycle waveform (provisional output waveform) at each point in time (when the output data 325 (see Figures 20 and 22) is a vector), then among the M peak positions (peak times) of the provisional output waveforms, peak positions where the confidence level (average value, etc.) for the period in the vicinity of each peak position is less than or equal to a predetermined threshold (e.g., 50%) may be excluded from the accumulation. Then, the integrated peak position may be determined based only on the peak positions where the confidence level (average value, etc.) for the period in the vicinity of each peak position is equal to or equal to a predetermined threshold (e.g., 50%). Alternatively, the confidence levels of the M provisional output waveforms may be added together at each point in time to obtain an average confidence level, and the integrated peak position may be determined based only on the peak positions within the period in which the average confidence level is equal to or equal to a predetermined threshold.

[0256] <4. Fourth Embodiment> In this fourth embodiment, a technique for estimating a heart rate cycle waveform based on a signal (complex signal, etc.) relating to a reflected wave at a certain spatial location will be described. Specifically, an oscillatory waveform (heart rate cycle waveform) including the fundamental frequency components of the heartbeat is reconstructed based solely on a complex signal relating to a single combination (single spatial location) Cn(r, θ). This technique will be explained with reference to Figure 26, etc. Figure 26 is a flowchart showing the process of the fourth embodiment. This process is performed by a controller 31, etc.

[0257] First, in step S51, the controller 31 acquires a complex signal relating to a single combination Cn(r, θ).

[0258] Next, in step S52, the controller 31 applies a high-pass filter having a predetermined cutoff frequency to the complex signal (more specifically, the time change Φ(τ) of the phase of the complex signal) (curve L11 (see Figure 8)) to generate a first data sequence (curve L21) mainly containing the harmonic components of the living body's heartbeat. In other words, an oscillatory wave mainly containing the harmonic components of the living body's heartbeat is extracted from the complex signal.

[0259] The upper part of Figure 27 shows a curve L21 relating to a certain spatial position (see also the right side of Figure 8). Curve L21 shows the complex signal after removing the respiratory component (including the main high-frequency components of respiratory) using a high-pass filter (specifically, the time change Φ(τ) of the phase Φ of the complex signal). The data sequence showing curve L21 corresponds to the first data sequence. Note that in Figure 27, the signal Φ(τ) over a 30-second period is shown.

[0260] Furthermore, in step S53, the controller 31 generates a second data sequence by applying an absolute value filter to the first data sequence. Specifically, the absolute value of the value at each time point τ is used as the value after applying the absolute value filter. This allows both positive and negative vibration components to be treated equally.

[0261] Furthermore, in step S54, the controller 31 applies a median filter to the second data column to generate a third data column. Specifically, the values ​​obtained by converting the second data column to the median within a predetermined period including each time point τ are used as the values ​​of each time point τ in the third data column after the median filter has been applied.

[0262] In detail, for example, the median value (the median of 20 values ​​from the 1st to the 20th frame) of the second data column within a predetermined period including the 1st frame (the period from the 1st frame to the 20th frame) is determined as the value for the 1st frame of the third data column. Similarly, the median value (the median of 20 values ​​from the 2nd to the 21st frame) of the second data column within a predetermined period including the 2nd frame (the period from the 2nd frame to the 21st frame) is determined as the value for the 2nd frame of the third data column. Likewise, the median value (the median of 20 values ​​from the 1st to the 20th frame) of the second data column within a predetermined period including the ith frame (the period from the ith frame to the (i+20th) frame) is determined as the value for the ith frame of the third data column. In this way, the third data column is generated.

[0263] By applying a median filter, the median value over a predetermined period (a predetermined time range) is obtained, thus suppressing abrupt changes. In other words, the third data column is obtained as a data column with abrupt changes suppressed compared to the second data column. This median filter is a type of smoothing filter. However, compared to applying an average filter (instead of a median filter) as a smoothing filter, it is possible to suppress the effects of outliers (noise, etc.).

[0264] Next, in step S55, the controller 31 applies an average value filter to the third data column to generate the fourth data column. Specifically, the values ​​obtained by converting the third data column to the average value over a predetermined period including each time point τ are used as the values ​​of each time point τ in the fourth data column after applying the average value filter. For example, the average value over a predetermined period including the i-th frame of the third data column (the period from the i-th frame to the (i+20)-th frame) (the average of the 20 values ​​from the i-th frame to the (i+20)-th frame) is determined as the value of the i-th frame in the fourth data column.

[0265] By applying an average value filter, the average value over a predetermined period (a predetermined time range) is obtained, which reduces the unevenness of the values ​​in the data column (time-series curve) after application, thus forming a smoother curve. In this way, applying an average value filter makes it easier to find the maximum or minimum value within a predetermined time range.

[0266] This fourth data sequence is generated as the biological heart rate cycle waveform (curve L61) (see the lower part of Figure 27). In the lower part of Figure 27, curve L21 and the heart rate cycle waveform (curve L61) generated based on curve L21 are shown together.

[0267] Furthermore, based on the obtained heart rate cycle waveform (curve L61), multiple peaks, multiple peak intervals, and representative peak interval values ​​(heart rate) are detected.

[0268] As described above, a first data sequence mainly containing the harmonic components of a living heartbeat is generated by applying a high-pass filter to a complex signal. Then, a second data sequence is generated by applying an absolute value filter to the first data sequence, and a third data sequence is generated by applying a median filter to the second data sequence, thereby generating the heart rate cycle waveform. Therefore, it is possible to easily generate the heart rate cycle waveform of a living organism.

[0269] Furthermore, by applying an average value filter to the third data column, a fourth data column is generated as a heart cycle waveform, making it possible to generate a smooth curve that facilitates the determination of peaks in the heart cycle waveform of a living organism.

[0270] In the fourth embodiment, the fourth data sequence is generated as a heart rate cycle waveform, but this is not limited to that. For example, the third data sequence may be generated as a heart rate cycle waveform. In other words, the heart rate cycle waveform may be generated without applying an average value filter.

[0271] <5. Fifth Embodiment> The fifth embodiment is a modification of the first embodiment, etc. The differences from the first embodiment, etc. will be explained in detail.

[0272] In this fifth embodiment, the heart rate period waveform (curve L61) reconstructed based on the time change Φ(τ) of the phase Φ of the complex signal for each combination (each spatial position) Cn(r, θ) according to the concept of the fourth embodiment is used as each of the N data sequences input to the learning model 410. In other words, a provisional heart rate period waveform (provisional heart rate period waveform) based on an oscillatory wave mainly containing the harmonic components of the heartbeat is used as each of the N data sequences. That is, N provisional heart rate period waveforms are adopted as input data 311D (see Figure 13).

[0273] In the fifth embodiment, a trained model 410 (420) is generated using training data consisting of N provisional heart rate cycle waveforms (curves L61) and the above-mentioned ground truth data of heart rate cycle waveforms (PPG waveforms, etc.).

[0274] Then, N provisional heart rate cycle waveforms relating to the person to be estimated are input to the trained model 420. More specifically, according to the concept of the fourth embodiment, N heart rate cycle waveforms (N curves L61) reconstructed based on N complex signals (for example, the time variation Φ(τ) of phase Φ) relating to N spatial positions Cn(r,θ) are input to the trained model 420. The output data output from the trained model 420 is then generated as the heart rate cycle waveform (curve L71) of the person to be estimated.

[0275] Curve L71 in Figure 28 is a diagram showing the estimated heart rate cycle waveform of such a subject, specifically the output data 315D from the trained model 420 (see Figure 13). In Figure 28, the heart rate cycle waveform measured at the same point using a PPG sensor (curve L81) is also shown (for comparison). Curve L71 is obtained as a heart rate cycle waveform that is close to the correct curve L81. In particular, the peak interval in curve L71 is very close to the peak interval in curve L81.

[0276] As shown in Figure 28, the estimation process using the learning model 410 yields a heart rate cycle waveform (estimated heart rate cycle waveform) (curve L71) that is close to the true heart rate cycle waveform (curve L81). In detail, although there are subtle differences in waveform shape between the estimated heart rate cycle waveform and the true heart rate cycle waveform, the intervals between each peak of the estimated heart rate cycle waveform are particularly close to the intervals between each peak of the true heart rate cycle waveform. Therefore, the representative peak interval (heart rate) can also be estimated well as a value close to the true value.

[0277] Furthermore, by adopting the N provisional heart rate cycle waveforms described above as input data for the learning model 410, it is possible to construct a good learning model 410 (trained model 420) based on a relatively small amount of training data (efficiently train the learning model 410). In short, it is possible to obtain heart rate cycle waveforms with a smaller amount of training data.

[0278] In the fifth embodiment, a provisional heart rate period waveform (provisional heart rate period waveform) based on an oscillatory wave (an oscillatory wave extracted from a complex signal) mainly containing the harmonic components of the heartbeat is adopted, which is a heart rate period waveform provisionally reconstructed using a high-pass filter, an absolute value filter, a median filter (and an average value filter), as in the fourth embodiment, etc. However, it is not limited to this. For example, the provisional heart rate period waveform may be a heart rate period waveform to which only a high-pass filter has been applied. Alternatively, the provisional heart rate period waveform may be a heart rate period waveform reconstructed by obtaining the envelope of the signal waveform after applying a high-pass filter. Alternatively, the waveform obtained by averaging the oscillatory wave waveform using a moving average and then differentiating it with respect to time may be adopted as the provisional heart rate period waveform.

[0279] <6. Sixth Embodiment> The sixth embodiment is a modification of the first embodiment, etc. The following will focus on the differences from the first embodiment.

[0280] In the first embodiment described above, the learning model 410 is trained to output a heart rate cycle waveform when N data sequences (such as the time change in the phase of a complex signal Φn(τ) (n=1,...,N)) relating to N spatial positions Cn(r,θ) are input (see upper part of Figure 10). Then, the heart rate cycle waveform of the target person is estimated using the trained learning model 410 (see lower part of Figure 10).

[0281] On the other hand, in this sixth embodiment, as shown in the upper part of Figure 29, etc., when N data sequences relating to N spatial locations are input, the learning model 410 (also referred to as 410F) is trained to output the "optimal location" (the optimal location from which the best data relating to a person's heart rate can be obtained among the multiple spatial locations). Then, the optimal location for data acquisition for the person to be estimated is estimated using the trained learning model 410 (also referred to as 420F) (see the lower part of Figure 29). In short, it differs from the first embodiment in that the "optimal location" (not the heart rate cycle waveform) is output (estimated). Furthermore, the heart rate cycle waveform of the person to be estimated is obtained using the data sequence relating to the optimal location, based on the concept of the fourth embodiment. Figure 29 is a diagram showing the learning stage processing and the inference stage processing according to the sixth embodiment.

[0282] The heart rate measurement device 10 according to the sixth embodiment is a measurement device for obtaining the optimal position from which the best heart rate data can be acquired. Once the optimal position from which the best heart rate data of the estimated target person can be acquired is known, it is possible to appropriately obtain the heart rate cycle waveform of the estimated target person based on the data from that optimal position. Consequently, it is possible to obtain a more accurate heart rate cycle waveform. Specifically, even when determining the heart rate cycle waveform based on a biosignal from only one specific location, it is possible to determine the heart rate cycle waveform more accurately by using the biosignal from the optimal position.

[0283] Figure 30 shows an example of the learning model 410F. The learning model 410F (also referred to as 510) in Figure 30 is a deep neural network model and has an input layer 511, an encoder 512, and an output layer 515.

[0284] The input data 511D input to the input layer 511 is the same as the input data 311D in the first embodiment and the like. The input data 511D has a structure (multi-channel structure) in which data sequences (one-dimensional array data consisting of 240 elements) relating to each complex signal of N spatial positions Cn(r,θ) (n=1,...,N) are arranged in individual channels on a data sequence basis (for each data sequence).

[0285] The encoder 512 is composed of multiple layers. In each layer, normalization processing (such as batch normalization processing) and convolution processing are performed. Data for the output layer 515 is generated by repeatedly performing convolution processing (and normalization processing, etc.) on the input data 511D input to the input layer 511 in the encoder 512.

[0286] The output data 515D output from the output layer 515 is output as data indicating the optimal position. The output data 515D has a size of 1 × 192 (= 1 × N).

[0287] A loss function F6 (for example, cross-entropy) is calculated for the difference (loss) between the output data 515D and the ground truth data 518 at the optimal position. The learning model 410F (specifically its parameters, etc.) is then trained to minimize (optimize) this loss function F6 (so that the output data 515D approaches the ground truth data 518).

[0288] Here, the "optimal position" is the position among N spatial positions that is most suitable for acquiring the best data (the best possible data) regarding a person's heart rate (the spatial position from which the best signal quality can be obtained). In other words, the data at the optimal position is the best data (the best possible data) regarding a person's heart rate among N data points.

[0289] The N spatial positions are represented by the element numbers of a row vector (1 × N) obtained by rearranging each element of a two-dimensional (or three-dimensional) array, indicated by distance r and angle θ (ψ), in a predetermined order in one dimension.

[0290] The correct data 518 for the optimal position is a 1 × N size vector (row vector). Of the N elements of this correct data 518 (row vector), only the value of the element corresponding to the optimal position is "1", and the values ​​of the elements corresponding to positions other than the optimal position are "0".

[0291] During training, this kind of correct answer data 518 is used.

[0292] Furthermore, during training, time-scale conversion and channel shuffling are performed, similar to the first embodiment. When performing channel drop, it is preferable not to drop the optimal position (the position with the best signal quality). Alternatively, the position with the best signal quality among the remaining data sequence after drop may be assigned as the ground truth data.

[0293] On the other hand, during inference, the number of the element corresponding to the highest value among the N elements of the output data 515D is determined as the optimal position.

[0294] Furthermore, the optimal position (optimal position for learning) in the correct data 518 is determined as follows (see Figures 31 and 32). Figures 31 and 32 illustrate the method for determining the correct data for the "optimal position" in the sixth embodiment.

[0295] First, the concept of the fourth embodiment is applied to each of the signals for N spatial positions (N combinations Cn(r,θ)) to obtain the heart rate period waveform. Specifically, a high-pass filter, absolute value filter, median filter, etc. are applied to the time change Φ(τ) of the phase of the complex signal for each combination Cn(r,θ) (each spatial position) (see Figure 26) to obtain the heart rate period waveform.

[0296] Figure 31 shows N heart rate cycle waveforms and the like obtained based on the method of the fourth embodiment.

[0297] The curve L21 on the left side of Figure 31 shows the time change Φ(τ) of the phase Φ of the complex signal after removing the respiratory component (including the main high-frequency components of respiration) using a high-pass filter. In other words, the signal in the graph on the left side of Figure 31 corresponds to the signal in the upper part of Figure 27. Note that two curves L21 out of 192 (N) are shown in the upper and lower parts of Figure 31.

[0298] On the other hand, the curve L61 on the right side of Figure 31 is the heart rate cycle waveform obtained by applying an absolute value filter, a median filter, etc., to the signal of curve L21. In other words, the signal in the graph on the right side of Figure 31 corresponds to the signal of curve L61 in the lower part of Figure 27.

[0299] Such curves L61 can be determined for each of the N spatial positions Cn, and a total of N curves L61 can be obtained (see the right side of Figure 31 and the left side of Figure 32).

[0300] Next, as shown in Figure 32, the correlation coefficient between each curve L61 and curve L81 is calculated. Curve L81 is a signal waveform that shows the correct heart rate cycle waveform, and is the heart rate cycle waveform measured at the earlobe at the same point using a PPG sensor.

[0301] However, there may be a time lag between the pulse wave measured by PPG (or the electrocardiogram waveform measured by ECG) and the waveform measured using the millimeter-wave sensor 20 (heart rate cycle waveform). Therefore, it is preferable to obtain several correlation coefficients between curves L61 and L81 after shifting the two curves L61 and L81 by a small amount of time relative to each other. It is also preferable to obtain the correlation coefficients between the two curves L61 and L81 in several states with different small time shifts, while gradually changing the shift period length (length of the small time). In this way, multiple correlation coefficients are calculated for curve L61 at each spatial position Cn. The maximum value among these multiple correlation coefficients is then calculated as the correlation coefficient for each spatial position Cn (maximum correlation coefficient for each spatial position). Furthermore, by obtaining such correlation coefficients for each spatial position Cn (maximum correlation coefficient for each spatial position) for N locations, the correlation coefficients for N locations can be obtained.

[0302] Of these N correlation coefficients (the largest correlation coefficient among the N locations), the spatial location corresponding to the largest correlation coefficient is determined as the ground truth data for the optimal location of the person being measured. In the ground truth data 518, only the element corresponding to the optimal location is set to "1," while the other elements are set to "0 (zero)." For example, if the 51st spatial location in a certain dataset (a certain N spatial locations Cn) is the optimal location, then in the ground truth data 518, which is represented by a 192-dimensional row vector, only the 51st element is "1," and the other elements (elements 1 to 50 and 52 to 192) are "0." The same process can be used to find the ground truth data for the optimal location (for example, the 38th location) for other datasets (another N spatial locations Cn).

[0303] The learning model 410F is trained using this ground truth data 518. Then, inference processing is performed using the trained model 420F. Specifically, the trained model 420F is input with N data sequences about the subject being measured (e.g., the time change in the phase of a complex signal Φn(τ) (n=1,...,N)), and the trained model 420 outputs output data 515D. The number of the element corresponding to the highest value (the value closest to "1") among the N elements of the output data 515D is then determined as the optimal position.

[0304] In this way, the controller 31 estimates the optimal position based on the trained model 420F.

[0305] Furthermore, the controller 31 determines the heart rate cycle waveform based on the signal at the optimal position (the time change Φ(τ) of the phase Φ of the complex signal). For example, the heart rate cycle waveform (curve L61) can be determined in the same manner as in the fourth embodiment. Then, based on this heart rate cycle waveform, peak detection processing, peak interval detection processing, and heart rate calculation processing can be performed.

[0306] As described above, a learning model 410 (trained model 420) is obtained that takes "multiple data sequences" based on multiple signals relating to multiple spatial locations near a single living organism as input and outputs the "optimal location" from which the optimal data relating to the heartbeat of the single living organism can be obtained among the multiple spatial locations. When "multiple data sequences" based on multiple signals relating to multiple spatial locations near a single living organism (estimated subject) are input to the learning model 410 (trained model 420), the "optimal location" from which the optimal data relating to the heartbeat of the single living organism can be obtained among the multiple spatial locations is estimated. According to this, even when determining the heart rate cycle waveform based on a biological signal from only one specific location, it is possible to determine the heart rate cycle waveform more accurately by using the biological signal from the optimal location.

[0307] Furthermore, by performing subsequent processing (processing in the fourth embodiment, etc.) based on the heart rate cycle waveform (curve L61) obtained from the signal at the optimal position (via the learning model 410), it becomes possible to more accurately determine the heart rate cycle waveform, peak position, peak interval, and heart rate.

[0308] <7. Seventh Embodiment> The seventh embodiment is a modification of the sixth embodiment. The following will focus on the differences from the sixth embodiment.

[0309] Figure 33 shows the processing steps for the learning stage and the inference stage of the seventh embodiment.

[0310] In the seventh embodiment, the learning model 410 (also referred to as 410G), shown in the upper part of Figure 33, is trained. This learning model 410G is a model that, when given N data sequences relating to N spatial locations as input, outputs both the "optimal location" and the "heart rate cycle waveform". During inference, the trained learning model 410 (also referred to as 420G) is used to estimate the optimal location for data acquisition for the target person (the optimal location out of N spatial locations) and the heart rate cycle waveform (also referred to as the first heart rate cycle waveform) based on the input data (N data sequences) (see the lower part of Figure 33). This differs from the sixth embodiment in that not only the "optimal location" but also the "heart rate cycle waveform" is output (estimated).

[0311] By learning not only the "optimal position" but also the "heart rate cycle waveform," information that contributes to the estimation of the optimal position can be obtained. Therefore, it is possible to improve the learning efficiency of the learning model 410, especially the learning efficiency related to the estimation of the optimal position, (particularly when the amount of data is small).

[0312] Subsequently, (similar to the sixth embodiment), the heart rate cycle waveform (also referred to as the second heart rate cycle waveform) of the estimated target person is determined based on the concept of the fourth embodiment using the data sequence relating to the optimal position. Furthermore, in the seventh embodiment, peak detection processing, peak interval calculation processing, and representative peak interval (heart rate) calculation processing are performed based on each of these two types of heart rate cycle waveforms (first heart rate cycle waveform and second heart rate cycle waveform).

[0313] Figure 33 is a diagram showing the processing of the learning stage and the inference stage according to the seventh embodiment, and Figure 34 is a diagram showing an example of the learning model 410G (also referred to as 520).

[0314] As shown in Figure 34, the learning model 410G of the seventh embodiment has a structure that combines the learning model 410 (310, etc.) of the first embodiment (see Figure 13) and the learning model 410F (510) of the sixth embodiment (see Figure 30).

[0315] The ground truth data for the optimal position is the same as in the sixth embodiment. The learning model 410G is trained using training data consisting of N data sequences and the ground truth data. At that time, the learning model 410G should be trained to minimize the loss function F7, which is expressed as a linear sum (for example, F6 + F1 × α2) of the loss function F1 related to the difference between the output data 315D and the ground truth data 318 of the heart rate cycle waveform and the loss function F6 described above. Alternatively, only the parameters of the hourglass-shaped network 310 after training may be learned first, and the learning of the parameters of the subnetwork 510 (510G) may be started with these parameters applied. Here, the value α2 is a parameter that adjusts the amount of heart rate cycle waveform learned in order to efficiently learn the optimal position.

[0316] Subsequently, inference processing is performed using the trained model 410G (also referred to as 420G). Specifically, input data 311D (in detail, N data sequences (time change Φn(τ) of the phase of a complex signal (n=1,...,N))) related to the subject of measurement (the subject of inference) is input to the trained model 420G, and output data 515D is output from the trained model 420G. The number of the element corresponding to the highest value among the N elements of the output data 515D is then determined as the optimal position.

[0317] Furthermore, the trained model 420G also outputs output data 315D for the input data 311D concerning the subject of measurement (the subject of inference). The output data 315D is data showing the heart rate cycle waveform. The heart rate cycle waveform directly output from the trained model 420G is also called the first heart rate cycle waveform (or model estimated waveform).

[0318] Furthermore, the heart rate cycle waveform is determined based on the "optimal position" signal (the time variation Φ(τ) of the phase Φ of the complex signal) (curve L21). For example, the heart rate cycle waveform is determined in the same manner as in the fourth embodiment. The heart rate cycle waveform obtained based on the optimal position signal (curve L21) is also referred to as the second heart rate cycle waveform (or optimal position waveform).

[0319] Based on the first heart cycle waveform, multiple peaks are detected, multiple peak intervals are calculated, and a representative peak interval (heart rate) is calculated. Similarly, based on the second heart cycle waveform, multiple peaks are detected, multiple peak intervals are calculated, and a representative peak interval (heart rate) is calculated.

[0320] Furthermore, the reliability of the inference results (heart rate cycle waveforms and optimal position (especially the heart rate cycle waveforms)) of the trained model 420G is determined based on whether the difference between the representative peak intervals based on these two types of heart rate cycle waveforms is below a certain level. Specifically, if the difference between the representative peak interval based on the first heart rate cycle waveform and the representative peak interval based on the second heart rate cycle waveform is smaller than a predetermined level, the inference results of the trained model 420G are determined to have a certain level of reliability or higher. Conversely, if the difference is larger than a predetermined level, the inference results of the trained model 420G are determined to not have a certain level of reliability or higher. In this way, it is possible to check the reliability of the inference results.

[0321] According to the above-described embodiment, it is possible to obtain the same effects as in the sixth embodiment.

[0322] Furthermore, the learning model 410 of the seventh embodiment is machine-learned to estimate not only the optimal position but also the heart rate cycle waveform. This means that information contributing to the estimation of the optimal position can be obtained through learning for the estimation of the heart rate cycle waveform. Therefore, it is possible to improve the learning efficiency of the learning model 410, particularly the learning efficiency related to the estimation of the optimal position, especially when the amount of data is small.

[0323] In the seventh embodiment described above, the learning model 410G (Figure 34) is composed of a combination of the hourglass-shaped network model 310 (see Figure 13) and the learning model 410F of the sixth embodiment (see Figure 30), but is not limited to this. For example, the learning model 410G may be composed of a combination of the diffusion model 350 (Figure 21) and the learning model 410F of the sixth embodiment.

[0324] <8. Eighth Embodiment> <Outline> In the eighth embodiment, as in the first embodiment, the heart rate cycle waveform is estimated without estimating the optimal position. Specifically, the learning model 410 is trained to output a heart rate cycle waveform when N data sequences relating to N spatial positions Cn(r,θ) are input. More specifically, each of the N data sequences is a data sequence (also called a phase data sequence) that shows the time change Φn(τ) (n=1,...,N) of the phase of a complex signal at each spatial position Cn. More specifically, as the data sequence showing the time change of the phase of the complex signal, a data sequence (see curve L21 in Figure 8) that shows the time change of the phase of the complex signal after applying a high-pass filter (harmonic components of the time change of the phase) is used. Then, the heart rate cycle waveform of the target person is estimated using the trained learning model 410.

[0325] However, the learning model 410 (also referred to as 410H) of the eighth embodiment (see Figure 36) includes a learning model 610 (also referred to as 610A) in addition to the learning model 810 similar to the learning model 410 of the first embodiment (more specifically, learning model 310 (see Figure 13)). The learning model 610A is provided in front of (preceding) the learning model 810. In the learning model 610A, a process is performed in which multiple data sequences, each showing the time change of the phase of a complex signal, are convolved at the same spatial position.

[0326] Specifically, the learning model 610A has a layer (convolutional layer) that convolves the multiple data sequences at the same spatial position. More specifically, the learning model 610A has a convolutional layer 630 (also referred to as 630A) that performs convolution processing for each of the N data sequences (more specifically, N phase data sequences), individually for each of the N data sequences.

[0327] Each convolutional layer 630 (630A) effectively extracts the characteristics of the time evolution of the complex signal phase at each spatial position. Specifically, it is possible to extract the characteristics of the periodic oscillation waveform corresponding to the heartbeat. Consequently, it is possible to improve the accuracy of estimating the heartbeat period waveform.

[0328] Furthermore, in the eighth embodiment, a broad predetermined region 110e (see the thick black rectangle in Figure 35), which includes both the predetermined regions 110a and 110b (see also Figure 6), is set as the measurement target region. This makes it possible to estimate the heart rate cycle waveform of a person within a broad measurement target region using only one predetermined region 110e, without switching between multiple predetermined regions 110a, 110b, etc., when the position of the human body (such as where a person stands) can change within a predetermined range (for example, a range including both "40 cm" and "80 cm"). The number of data sequences N in the eighth embodiment is greater than the number of data sequences in the first embodiment (for example, 192) (for example, 456). Figure 35 is a diagram showing the measurement target region, etc., in the eighth embodiment (and the ninth embodiment).

[0329] In detail, the area to be measured is defined as an area where the probability of a person performing a certain action (specifically, the area around the person's chest) being present is higher than a predetermined level. For example, the area where a person operating an Automatic Teller Machine (ATM) is likely to be present (an area encompassing the area around the chest of a person standing at any position within a predetermined range in front of the ATM (e.g., a distance of "20 cm" to "90 cm" from the ATM)) is defined as the predetermined area 110e (see Figure 35). The predetermined area 110e is composed of multiple (N; here, N = 456) divided areas (rectangular areas) divided into, for example, 24 × 19 (456 in total) (divided into 24 in the angle θ direction and 19 in the distance r direction). The predetermined area 110e is defined as an area ranging from +30 degrees to approximately -18 degrees in angle θ and from approximately 50 cm to approximately 150 cm in distance r. According to this, it is possible to detect operators (for example, victims of special fraud) who have an elevated heart rate in front of ATMs, etc.

[0330] In addition, the same process of estimating the heart rate cycle waveform for a person (such as an ATM operator) can be performed in the learning model 410 of the first embodiment. However, in the learning model 410 of the first embodiment, this is done by switching between predetermined regions 110a and 110b (see Figure 6) according to the detailed position of the person. In contrast, in this eighth embodiment, instead of switching between multiple predetermined regions 110a and 110b, one (relatively broad) predetermined region 110e is always used.

[0331] In particular, the learning model 410, which includes a convolutional layer 630 (a convolutional layer that performs convolutional processing on the sequence of phase data at each spatial location), can effectively extract the characteristics of the time change of the phase of the complex signal at each spatial location (such as the characteristics of the periodic oscillation waveform corresponding to the heartbeat). Specifically, the machine learning can be performed in a way that emphasizes the characteristics of phase changes (signal changes representing the heartbeat) at locations where a person is present, and disregards phase signals (noise that does not represent the heartbeat) at locations where a person is not actually present. Consequently, even when the measurement target space is extensive and includes locations where no person is present (locations where a person is not actually present), accurate inference is possible. Therefore, it is not necessary to switch between predetermined regions 110a, 110b, etc., depending on the position of the operator (person being measured), and it is possible to accurately estimate the heartbeat period waveform, etc., for a person within a wide measurement target area using only one predetermined region 110e.

[0332] <Learning Model 410H According to the Eighth Embodiment> The learning model 410H according to the eighth embodiment is configured to include two learning models (also referred to as submodels or sub-learning models) 610A and 810 (see Figure 36). In detail, the learning model 410H includes a submodel 610A in the front stage (upper side in Figure 36) and a submodel 810 in the rear stage (lower side in Figure 36).

[0333] The subsequent submodel 810 has the same configuration as the learning model 410 (also referred to as 410A) according to the first embodiment, more specifically the hourglass network model 310 (see Figure 13, etc.). The submodel 810 is a deep neural network model (hourglass network model) comprising an input layer, an encoder with downsampling, a bottleneck layer, a decoder with upsampling, and an output layer.

[0334] The output of the preceding submodel 610 is used as the input to the subsequent submodel 810. More specifically, the new N data sequences after feature extraction, output from the preceding submodel 610, are used as multiple (N) channel data inputs to the subsequent submodel 810.

[0335] The preceding submodel 610 (610A) receives input data 311D, similar to the learning model 410 in the first embodiment. In other words, N data sequences representing the time change Φn(τ) of the complex signal phase at each spatial position are input (n=1,...,N). Specifically, similar to the first embodiment, the data sequence after applying a high-pass filter to the phase signal (which is also one of the provisionally reconstructed provisional heart rate period waveforms) is used as input data for the learning model 410H (and submodel 610). However, it is not limited to this, and, similar to the fifth embodiment, a data sequence to which absolute value filters and median filters have also been applied may be used as input data for the learning model 410H. Alternatively, the data sequence before applying the high-pass filter (the data sequence of the phase signal) may be used as input data for the learning model 410H.

[0336] Submodel 610A has multiple (N) convolutional layer groups 630 (630A). More specifically, submodel 610 has a multi-layer structure (here, a 5-layer multilayer structure) (also called a multi-stage structure) of convolutional layer groups 630A for each of the multiple (N) spatial positions (in other words, for each of the N data sequences). Thus, submodel 610A comprises N convolutional layer groups (also simply called convolutional layers) 630A. Each convolutional layer 630A can also be described as a (multi-stage) convolutional layer (multi-stage convolutional layer) that performs a convolution process in multiple stages on a data sequence that shows the time change of a complex signal (phase, etc.) at each spatial position.

[0337] As described above, input data 311D is input to submodel 610A. Here, input data 311D is composed of N-column vector data (N data columns) consisting of 240 data (elements) per column.

[0338] In detail, for each of the multiple (N) convolutional layers 630A (of the submodel 610A), one data sequence from among the N data sequences (input data 311D) (here, a data sequence showing the time change Φ(τ) of the phase Φ of a complex signal at a single spatial position) is input. Specifically, the nth data sequence from among the N data sequences is input to the nth convolutional layer 630A of the N convolutional layers 630A (n=1, ..., N).

[0339] Then, in each convolutional layer 630A, two or more data sequences are generated that represent the time change Φn(τ) of the phase of a complex signal at the same spatial position, and a convolution process is performed on these two or more data sequences.

[0340] Specifically, for example, in the first convolutional layer (group of convolutional layers) 630A (n=1), the convolution process by the initial convolutional layer 632 (Figure 36) generates two or more (four in this case) data sequences that represent the time change Φn(τ) of the phase of the complex signal at the same spatial position. Specifically, first, a convolution process is performed on one data sequence (for example, the first data sequence (n=1)) using two or more (four in this case) different kernels (one-dimensional kernels) (shown as four short rectangles in Figure 36), thereby generating a data sequence with two or more (four in this case) channels. In other words, by performing four separate convolution processes on one data sequence (for example, 240 × 1 size data) by the convolutional layer 632, a four-channel data sequence (for example, 240 × 4 size data (data with four channels)) is generated from that one data sequence. The data sequences of the four channels are (two or more) data sequences that show the time variation Φn(τ) of the phase of a complex signal at the same spatial location.

[0341] Then, a convolution process (one-dimensional (and four-channel) convolution process) is performed on the (original) four-channel data sequence (a data sequence of two or more) by the convolutional layer 633. This generates a data sequence of two or more (four) channels (new four-channel data).

[0342] In the convolutional layer 633, a convolution operation is performed on a data set consisting of the original 4-channel data sequence using a kernel (a 1-dimensional (and 4-channel) kernel), generating one new channel data. A similar convolution operation is repeated (a total of 4 times) while changing the kernel (a 1-dimensional (and 4-channel) kernel), generating a total of 4 new channel data. As a result, the convolutional layer 633 generates new 4-channel data sequences from the original 4-channel data sequences.

[0343] Furthermore, the same processing as that performed in convolutional layer 633 is carried out by the subsequent convolutional layers 634, 635, and 636, respectively. Feature extraction is advanced through the convolutional processing by these convolutional layers, and a new updated 4-channel data sequence is sequentially generated, creating the (final) 4-channel data sequence. Then, further convolution processing is performed on this (final) 4-channel data sequence by one kernel (a 1-dimensional (and 4-channel) kernel) of convolutional layer 637, generating output data 639D (single-channel data) from convolutional layer 630A. Output data 639D can also be described as a new data sequence (phase data sequence after feature extraction) from which the features of the time change of phase have been extracted. Note that each kernel within convolutional layer 630 has different parameter values.

[0344] Similarly, for each of the other data sequences among the N data sequences (the nth data sequence; n = 2, ..., N), a new data sequence (a phase data sequence after feature extraction) is generated by extracting the features of the time change of the phase. Specifically, in the nth convolutional layer 630A, the nth data sequence is input, and two or more data sequences representing the time change Φn(τ) of the complex signal at the same spatial position are generated based on the nth data sequence (one data sequence), and a convolution process is performed on these two or more data sequences. Then, a phase data sequence (a new data sequence) 639D after feature extraction for the nth data sequence is generated.

[0345] In this way, each of the N convolutional layers 630A performs convolution on two or more data sequences (phase data sequences) at the same spatial location, generating each phase data sequence after feature extraction. Then, the N convolutional layers 630A (layers that convolve each of the N data sequences) generate N phase data sequences after feature extraction. As a result, the time-varying phase features of the complex signal are extracted for each spatial location.

[0346] These N feature-extracted phase data sequences are output from the preceding submodel 610A.

[0347] The output from the submodel 610A (the phase data sequence 639D after extracting N features, also referred to as the new N phase data sequence) is then used as input to the subsequent submodel 810. In other words, the new N phase data sequence 639D is used as multiple (N) channel data (for example, 240 × N size data) input to the input layer (and thus the encoder) of the submodel 810.

[0348] <Learning and Inference Stages (8th Embodiment)> The learning model 410H is machine-learned based on multiple training data (supervised data). In this embodiment, in particular, each training data consists of data at a relatively large number of spatial locations (for example, 456). The evaluation function (loss function) can be the same as that used in the first embodiment, etc. The learning model 410H after machine learning is generated by performing machine learning to optimize (minimize) the evaluation function. Specifically, the learning parameters of submodel 610 (parameters of each kernel, etc.) and submodel 810 are optimized by this machine learning.

[0349] Subsequently, inference processing is performed using the machine learning model 410H. This makes it possible to accurately infer the heart rate cycle waveform of the person being measured.

[0350] <Effects of the Eighth Embodiment> In this eighth embodiment, a convolution process is performed on the data sequence of each spatial location in each convolutional layer 630 of the submodel 610 of the learning model 410H. The convolution process of the convolutional layer 630 allows for the extraction of characteristics of the time change of the phase of the complex signal at each spatial location. Specifically, characteristics of the periodic oscillation waveform corresponding to the heartbeat can be extracted. More specifically, the machine learning is designed to emphasize the characteristics of the phase change (signal change representing the heartbeat) at the location where the person is present, and conversely, to disregard the phase signal (noise, etc., that does not represent the heartbeat) at the location where the person is not actually present. Therefore, even if the measurement target space includes both locations where the person is not actually present and locations where the person is actually present, the signal at the location where the person is actually present can be appropriately reflected in the estimated heartbeat period waveform. In short, it is possible to accurately infer the heartbeat period waveform even when the measurement target space includes areas where the person is absent.

[0351] In particular, in each convolutional layer 630, convolution is performed on two or more data sequences that represent the time evolution of the phase of a complex signal at the same spatial location, generating two or more data sequences. Therefore, various characteristics of the time evolution of the phase of a complex signal at each spatial location can be extracted effectively.

[0352] Furthermore, in each convolutional layer 630, multiple stages of convolution are performed on a data sequence representing the time evolution of the complex signal phase at each spatial position. Therefore, the characteristics of the time evolution of the complex signal phase at each spatial position can be extracted more effectively.

[0353] The learning model 410H, which includes a convolutional layer 630, makes it possible to improve inference accuracy. In particular, it is possible to perform accurate inference even when the measurement target space is extensive and includes locations where no person is present (locations where no person actually exists). Therefore, it is not necessary to switch between predetermined areas 110a, 110b, etc., depending on the position of the operator (person being measured), and it is possible to always use only one extensive predetermined area 110e as the measurement target area and accurately estimate the heart rate cycle waveform, etc., of the person within that extensive measurement target area (predetermined area 110e).

[0354] <First Modification of the Eighth Embodiment> In the eighth embodiment described above, two or more data sequences representing the time change Φn(τ) of the phase of the complex signal at the same spatial position are generated by the convolution process (of the convolutional layer 632) with respect to the nth data sequence. However, the present invention is not limited thereto.

[0355] For example, as shown in Figure 37, two or more data sequences representing the time change Φn(τ) of the complex signal's phase at a given spatial location may be generated by copying the nth data sequence (which represents the time change Φn(τ) of the complex signal's phase at a given spatial location). In the convolutional layer 630 (also referred to as 630B) of the learning model 410 (also referred to as 410J) in Figure 37, there is no convolutional layer 632 (see Figure 36). By copying one nth data sequence (n=1,...,N), two or more (for example, four) data sequences representing the time change Φn(τ) of the complex signal's phase at the same spatial location are generated. Then, convolution processing is performed on these two or more data sequences (by the convolutional layer 633, etc.).

[0356] Such a convolutional layer 630B may be provided.

[0357] <Second Modification of the Eighth Embodiment> In addition, although a one-dimensional convolution process is performed in the convolutional layer 630 of the eighth embodiment described above, the invention is not limited to this, and a two-dimensional convolution process may also be performed (see Figure 38). Figure 38 shows a modified example involving a two-dimensional convolution process.

[0358] In the modified learning model 410 (also referred to as 410K) shown in Figure 38, a convolutional layer 630 (also referred to as 630C) involving two-dimensional convolution is provided individually for each of the N data sequences.

[0359] In each convolutional layer 630C shown in Figure 38, a convolution process is performed on two or more phase data sequences (two or more identical phase data sequences showing the time change Φ(τ) of the complex signal at each spatial position) that are generated by duplicating the data sequence.

[0360] Here, a data column with six columns is given as an example of two or more data columns. The number of columns in the data column may be any other number (for example, two, three, four, five, seven, or eight columns). It is preferable that the data column has three or more columns.

[0361] Specifically, as shown in Figure 38, in each convolutional layer 630C, six phase data sequences are generated by duplicating a single data sequence that shows the time evolution of the complex signal at each spatial position Cn six times. However, the process is not limited to this, and the six phase data sequences may include the original phase data sequence itself.

[0362] Then, in each convolutional layer 630C, a convolution process is performed on the two or more (six) phase data sequences (six identical data sequences that show the time change of the phase).

[0363] As described above, submodel 610 has a multilayer convolutional layer 630C for each of the N data sequences (N spatial positions).

[0364] In each convolutional layer 630C, first, one of the N input data sequences (the nth data sequence, for example, the first data sequence (n=1)) is duplicated to generate six identical data sequences. Then, these six data sequences (six 240-dimensional vectors), each extending vertically, are arranged horizontally to form a two-dimensional structure (see the top row of Figure 38).

[0365] In Figure 38, the horizontal direction in each layer of the submodel 810 (see bottom row) represents the channel direction (similar to the learning model 410 in Figure 13), and the multiple vertical bars arranged horizontally in each layer of the submodel 810 in Figure 38 represent data sequences with multiple channels. For example, the multiple vertical bars labeled "240×48" represent a 240-dimensional vector with 48 channels (a 1-dimensional data sequence with 48 channels). This 240-dimensional vector with 48 channels can also be expressed as data with a size of "1×240×48". In other words, "240×48" is an expression that omits the "1×" at the beginning.

[0366] On the other hand, in the submodel 610 of Figure 38, each data column is displayed three-dimensionally, and the depth direction (diagonal direction) in each layer of the submodel 610 of Figure 38 does not indicate the channel direction, nor do multiple adjacent data columns indicate multiple channels. The six data columns arranged in the depth direction (diagonal direction) within the submodel 610 of Figure 38 (six data columns labeled "6×240 (×1)") are two-dimensionally arranged data columns (two-dimensional array data), and their number of channels Nc is "1". These six data columns are data with a size of 6×240×1 (a size with a channel count of "1"). Here, the size of these six data columns is also simply expressed as the size of "6×240", omitting the channel count of "1". The same applies to data columns with other numbers of columns in the submodel 610 ("5×240", "4×240", ..., "1×240", etc.).

[0367] Then, a two-dimensional convolution is performed on the six data columns arranged in two dimensions. Specifically, for every two adjacent columns of the six data columns (a data column of size 6 x 240), a two-dimensional convolution is performed using a kernel K of a predetermined size (for example, 7 x 2 size) (the first stage of two-dimensional convolution). The first stage of two-dimensional convolution generates a five-column data column (arranged in two dimensions) (a data column of size 5 x 240) for the next stage (second stage) based on the six data columns.

[0368] Specifically, first, the leftmost data column (the leftmost column) and the data column to its right (the second column from the left) of the six data columns (a data column of size 2 × 240 × 1) are treated as equivalent to a two-dimensional pixel arrangement, and a two-dimensional convolution operation is performed on these two data columns using a kernel K of a predetermined size (for example, 7 × 2). More specifically, in this two-dimensional convolution operation, the convolution operation using the 7 × 2 size kernel K is performed while sequentially shifting the kernel K downwards with a stride of 1, and 240 data points are generated with padding at both ends. Note that these two adjacent data columns are data of size 2 × 240 × 1 (in other words, data with "1" channel). This two-dimensional convolution operation is performed on the array data (2 × 240 size two-dimensional array data) with "1" channel. The result of this two-dimensional convolution process is output as the leftmost data column in the next stage's five-column data sequence (a data sequence with a size of 5 × 240 × 1).

[0369] Next, a two-dimensional convolution is performed on the two data columns (2x240x1 size) consisting of the second column from the left (the second column from the leftmost column) and the data column to its right (the third column from the left) of the six data columns, using a kernel K of a predetermined size (for example, 7x2 size). The result of this two-dimensional convolution is output as the second data column from the left of the five data columns (5x240x1 size) in the next stage.

[0370] Similarly, a two-dimensional convolution is performed on the third and fourth columns from the left of the six-column data sequence. The result of this two-dimensional convolution is output as the third column from the left of the five-column data sequence in the next stage.

[0371] Similarly, a two-dimensional convolution operation is performed on the fourth and fifth columns from the left of the six-column data sequence to generate the fourth column from the left of the five-column data sequence in the next stage. Furthermore, a two-dimensional convolution operation is performed on the fifth and sixth columns from the left of the six-column data sequence to generate the fifth column from the left of the five-column data sequence in the next stage.

[0372] In this way, convolution is sequentially performed on two adjacent columns of the total six phase data columns, generating a five-column data column for the next stage (second stage) from the six-column data of the first stage. This five-column data column can also be described as the data set (feature vector set) after feature extraction in the first stage.

[0373] Through a similar process, a 4-column data sequence (4x240 size) is generated from the 5-column data sequence (arranged two-dimensionally) (5x240 size data sequence) of the second stage.

[0374] Furthermore, through a similar process, a 3-column data sequence (a 2-dimensionally arranged, 4x240 size data sequence) is generated from the 4-column data sequence of the third stage (a 2-dimensionally arranged, 4x240 size data sequence) to the next stage (fourth stage). From this 3-column data sequence of the fourth stage, a 2-column data sequence (a 2-dimensionally arranged, 2x240 size data sequence) is generated to the next stage (fifth stage). Furthermore, from the 2-column data sequence of the fifth stage, a 1-column data sequence (a 1x240 size data sequence) (a 240-dimensional feature vector) is generated to the next stage (sixth stage).

[0375] This generates a data sequence (a 1x240 size data sequence) (a 240-dimensional vector) in which the time evolution characteristics of the phase are extracted for one of the N data sequences (the first data sequence).

[0376] Similarly, for each of the other data sequences among the N data sequences (the nth data sequence; n=2, ..., N), a new data sequence (the phase data sequence after feature extraction) is generated, from which the time evolution characteristics of the phase have been extracted.

[0377] In this way, each of the N convolutional layers 630C performs a convolution operation on two or more (identical) data sequences (phase data sequences) at each spatial position, generating each phase data sequence after feature extraction. Then, N feature-extracted phase data sequences are generated by the N convolutional layers 630C. As a result, the features of the time evolution of the phase of the complex signal are extracted for each spatial position.

[0378] These N feature-extracted phase data sequences are output from the preceding submodel 610.

[0379] The output from the submodel 610 (N phase data sequences after feature extraction, also referred to as new N phase data sequences) is then used as input to the subsequent submodel 810. More specifically, these new N phase data sequences are used as multiple channel data input to the input layer (and consequently the encoder) of the submodel 810.

[0380] In Figure 38, six phase data sequences are generated in each convolutional layer 630C by duplicating a single data sequence (a single phase data sequence) that shows the time change of the phase of a complex signal at each spatial position Cn six times, but the method is not limited to this. For example, in each convolutional layer 630C, six phase data sequences at the same spatial position may be generated by individually performing convolution processing with six different kernels on a single phase data sequence at a certain spatial position. <Third Modification of the Eighth Embodiment> In the eighth embodiment and the like, each convolutional layer 630 performs convolution processing on two or more data sequences that show the time change of the phase of a complex signal at the same spatial position, but the method is not limited to this. Specifically, each convolutional layer 630 may perform convolution processing on a single data sequence that shows the time change of the phase of a complex signal at a single spatial position.

[0381] For example, in each convolutional layer 630A (Figure 36), only a single (one-dimensional) convolution operation may be performed on a single phase data sequence (a single input data sequence) to generate only single-channel data. Furthermore, multiple stages of (one-dimensional) convolution operations may be performed on this single-channel data to generate output data 639D. The same applies to other convolutional layers 630B, etc.

[0382] <9. Ninth Embodiment> The ninth embodiment is a modification of the eighth embodiment. In the ninth embodiment, as in the eighth embodiment, a wide predetermined area 110e (see Figure 35) is set as the measurement target area. Without switching between multiple predetermined areas 110a, 110b, etc., the heart rate cycle waveform of a person within the wide measurement target area is estimated using only one predetermined area 110e. The number of data sequences N is greater than the number in the first embodiment (for example, 192) (for example, 456).

[0383] The following section will focus on explaining the differences from the eighth embodiment.

[0384] In the eighth embodiment described above, each of the multiple data sequences input to the learning model 400 consists only of data sequences that show the time change in the phase of the complex signal at each spatial position Cn.

[0385] On the other hand, in this ninth embodiment, each of the multiple (N) data sequences input to the learning model 400 (also referred to as 410L) is composed of both a data sequence showing the time change Φn(τ) of the "phase" of the complex signal at each spatial position Cn (n=1,...,N) and a data sequence showing the time change Ψn(τ) of the "norm" (intensity) of the complex signal. In short, multiple data sequences, each composed of both phase and norm, are used as input data 311D. The data sequences showing the time change Φn(τ) of the phase of the complex signal at N locations and the data sequences showing the time change Ψn(τ) of the norm of the complex signal are each generated, for example, as N (456) 240-dimensional vector data. A data sequence for a single spatial position consists of two vector data, one for phase and one for norm. In other words, a total of (N × 2) 240-dimensional vector data are generated as input data 311D.

[0386] The learning model 410 (410L) of the ninth embodiment is configured to include two learning models (also referred to as sub-models or sub-learning models) 710 and 810 (see Figure 39).

[0387] The learning model 410L is similar to the learning model 410H (see Figure 36). However, the learning model 410L differs from the learning model 410H in that it includes a submodel 710 instead of the submodel 610 of the learning model 410H. In the submodel 710, a process is performed in which multiple data sequences (each containing both phase and norm data sequences) are convolved at the same spatial position.

[0388] Submodel 710 comprises N convolutional layer groups 720. Each convolutional layer group 720 (also simply referred to as a convolutional layer 720) performs convolution based on a data sequence representing the time change Ψn(τ) (n=1,...,N) of the norm (intensity) of the complex signal at each spatial position Cn. Specifically, each convolutional layer 720 performs convolution on a data sequence representing the time change Φn(τ) of the phase of the complex signal at each spatial position Cn (phase data sequence) and a data sequence representing the time change Ψn(τ) of the norm of the complex signal (also referred to as a norm data sequence or intensity data sequence). Such a convolutional layer 720 is provided for each of the N data sequences (see Figure 39).

[0389] Furthermore, each convolutional layer 720 includes a phase convolutional layer 730 that performs phase-related convolution processing and a norm convolutional layer 740 that performs norm-related convolution processing. In the phase convolutional layer 730, N data sequences relating to phase are convolved at the same spatial position, and in the norm convolutional layer 740, N data sequences relating to norm are convolved at the same spatial position.

[0390] The phase convolutional layer 730 has the same configuration as the convolutional layer 630 described above (Figure 36). Specifically, the phase convolutional layer 730 comprises convolutional layers 732, 733, 734, 735, and 736. Convolutional layers 732, 733, 734, 735, and 736 are the same as convolutional layers 632, 633, 634, 635, and 636, respectively.

[0391] However, the output data 739D from the phase convolution layer 730 is not a single data sequence (single-channel data), but rather multiple channel data generated by several stages of convolution processing. Specifically, the phase convolution layer 730 outputs a data sequence (4-channel data) 739D immediately after the convolution processing by the convolution layer 736. In this way, each phase convolution layer 730 performs convolution processing on two or more data sequences that represent the time change of the phase of a complex signal at the same spatial position, generating a convolutional phase data sequence 739D which is a data sequence of two or more channels.

[0392] The norm convolutional layer 740 has the same configuration as the phase convolutional layer 730. However, the kernel in the norm convolutional layer 740 and the kernel in the phase convolutional layer 730 have different parameter values.

[0393] Furthermore, the norm convolution layer 740 outputs multiple channel data generated by several stages of convolution processing. In each norm convolution layer 740, convolution processing is performed on two or more data sequences that represent the time evolution of the norm of a complex signal at the same spatial position to generate a convolutional norm data sequence 749D which is a data sequence with two or more channels (for example, four channels).

[0394] Furthermore, each convolutional layer 720 also includes an integrated convolutional layer 750. The integrated convolutional layer 750 is a convolutional layer that integrates the output data (convolutional phase data) 739D from the phase convolutional layer 730 and the output data (convolutional norm data) 749D from the norm convolutional layer 740 to perform convolution processing.

[0395] In each integrated convolutional layer 750, convolution processing is also performed on data sequences of four or more channels, which are formed by integrating the convolutional phase data sequence 739D and the convolutional norm data sequence 749D. More specifically, in each integrated convolutional layer 750, convolution processing is also performed on data sequences of four or more channels (in this case, eight channels), which are formed by integrating the convolutional phase data sequence 739D of two or more channels (in this case, four channels) output from each convolutional layer 730A and the convolutional norm data sequence 749D of two or more channels (in this case, four channels) output from each convolutional layer 740A. For example, convolution processing using one kernel (a one-dimensional (and eight-channel) kernel) is performed eight times on the eight-channel data sequence, using eight different kernels, to generate a new eight-channel data sequence. Then, a convolution process is performed on the new 8-channel data sequence using a kernel (a one-dimensional (and 8-channel) kernel), generating a single data sequence (the integrated convolutional data sequence) 759D. This single data sequence 759D is output from each convolutional layer 750 (and consequently, each convolutional layer 720).

[0396] In this way, each of the N convolutional layers 720 performs convolution on two or more data sequences at the same spatial location, generating each data sequence after feature extraction. Then, the N convolutional layers 720 (layers that convolve each of the N data sequences) generate N data sequences 759D after feature extraction. As a result, the features of the time change in the phase of the complex signal are extracted for each spatial location, while reflecting the features of the time change in the norm of the complex signal.

[0397] These N data sequences 759D (after feature extraction) are output from the preceding submodel 710A.

[0398] The output data from the submodel 710A (the phase data sequence 759D after extracting N features, also referred to as the new N phase data sequence) is then used as input to the subsequent submodel 810. In other words, the new N phase data sequence 759D is used as multiple (N) channel data (for example, 240 × N size data) input to the input layer (and thus the encoder) of the submodel 810.

[0399] <Effects of the Ninth Embodiment> In each convolutional layer 720 of the ninth embodiment, convolution processing is performed on a data sequence relating to the phase of the complex signal at each spatial position, similar to each convolutional layer 630 of the eighth embodiment. This makes it possible to obtain the same effects as in the eighth embodiment, such as being able to extract well the characteristics of the time change of the phase of the complex signal at each spatial position.

[0400] Furthermore, from the following perspective, the ninth embodiment can achieve even greater effectiveness than the eighth embodiment. Specifically, in particular, in the N convolutional layers 720 of the ninth embodiment, convolution processing is performed not only on the phase of the complex signal but also on both the phase and norm of the complex signal (more specifically, convolution processing on a data sequence integrating the phase and norm). This allows for good extraction of the characteristics of the time change of the phase according to the norm (intensity) of the complex signal for each spatial position Cn. More specifically, the strength of the phase (phase signal) can be goodly adjusted by each convolutional layer 720 according to the norm (intensity signal). More specifically, the phase signal at positions with a large intensity signal (phase signal at positions representing heartbeats) is increased, and the phase signal at positions with a small intensity signal (noise signals, etc. (phase signals other than heartbeats)) can be attenuated. Therefore, even if multiple spatial positions Cn (measurement target space) include both positions where a person does not actually exist and positions where a person actually exists, the signal at the position where a person actually exists can be appropriately reflected in the heartbeat cycle waveform. In other words, it is possible to infer heart rate cycle waveforms with greater accuracy even when the measurement target space includes areas where no person is present. Therefore, it is not necessary to switch predetermined areas depending on the position of the person being measured, but rather to always use only one broad predetermined area as the measurement target area, and it is possible to estimate heart rate cycle waveforms and the like of the person within that measurement target area with greater accuracy.

[0401] For example, similar to the eighth embodiment described above, when detecting an operator (e.g., a victim of a special fraud) with an elevated heart rate in front of an ATM or the like, a relatively broad predetermined area 110e is set as the area where the operator (the person to be measured) is likely to be located. Then, without switching between predetermined areas 110a, 110b, etc., depending on the operator's position, it is possible to always use only this one predetermined area 110e to accurately estimate the heart rate cycle waveform, etc., of a person within the broad measurement target area. In particular, by performing convolution processing on both the phase and norm of the complex signal, it is possible to estimate the heart rate cycle waveform, etc., even more accurately.

[0402] <First Modification of the Ninth Embodiment> The ninth embodiment can also be modified in the same way as the eighth embodiment.

[0403] Figure 40 shows a learning model 410 (also referred to as 410M) according to a modified example of the ninth embodiment (also referred to as the first modified example).

[0404] For example, as shown in Figure 40, two or more data sequences showing the time change in phase Φn(τ) and the time change in norm Ψn(τ) of a complex signal at a certain spatial location may be generated by copying the nth data sequence (a data sequence showing the time change in phase Φn(τ) and the time change in norm Ψn(τ) of a complex signal at the same spatial location). In the convolutional layer 730 (also referred to as 730B) of the learning model 410 (410M) in Figure 40, the convolutional layer 732 (see Figure 39) is not provided. The same applies to the convolutional layer 740 (also referred to as 740B).

[0405] By copying the first nth data sequence (n=1, ..., N), two or more (for example, four) data sequences (convolutional phase data sequences) representing the time change Φn(τ) of the phase of the complex signal at the same spatial location, and two or more (for example, four) data sequences (convolutional norm data sequences) representing the time change Ψn(τ) of the norm of the complex signal at the same spatial location are generated. Then, convolution processing is performed on the two or more data sequences relating to the phase by the convolutional layer 730B, and convolution processing is performed on the two or more data sequences relating to the norm by the convolutional layer 740B.

[0406] Such convolutional layers 730B, 740B, etc., may be provided.

[0407] <Second Modification of the Ninth Embodiment> Figure 41 shows a learning model 410 (also referred to as 410P) relating to a further modification of the ninth embodiment (also referred to as the second modification).

[0408] The learning model 410P includes a submodel 710C as a submodel 710, and the submodel 710C includes a plurality of convolutional layers 720C (also referred to as 760) as a plurality of convolutional layers 720.

[0409] In each convolutional layer 760 of the learning model 410P, a convolution operation is performed on one of the N data sequences (the nth data sequence).

[0410] However, in each convolutional layer 760, for each spatial position, one data sequence is used that shows the time change Φn(τ) of the phase of the complex signal, and another data sequence that shows the time change Ψn(τ) of the norm.

[0411] Specifically, first, one data sequence representing the time evolution Φn(τ) of the phase of a complex signal at a given spatial location and another data sequence representing the time evolution Ψn(τ) of the norm of the complex signal at the same spatial location are integrated as two channels of data (in total).

[0412] Then, the convolution process on the data of the two channels is performed by multiple convolutional layers 763, 764, 765, and 766.

[0413] In the convolutional layer 763, four different kernels (each a one-dimensional (and two-channel) kernel) are used to generate four-channel data from the two-channel data.

[0414] In the convolutional layer 764, new 4-channel data is generated using four different kernels (each a 1-dimensional (and 4-channel) kernel) from the 4-channel data.

[0415] In the convolutional layer 765, new (latest) 4-channel data is generated in the same manner as in the convolutional layer 764.

[0416] In the convolutional layer 766, output data 769D (1-channel data) is generated using a single kernel (a 1-dimensional (and 4-channel) kernel) for the latest 4-channel data. This output data 769D is then output from each convolutional layer 760.

[0417] Then, the N output data (N data sequences) 769D from the N convolutional layers 760 are used as multiple (N) channel data inputs to the subsequent submodel 810.

[0418] A learning model 410P of this nature may be used.

[0419] <Third Modification of the Ninth Embodiment> Figure 42 shows a learning model 410 (also referred to as 410Q) relating to a further modification of the ninth embodiment (also referred to as the third modification).

[0420] The learning model 410Q includes a submodel 710D as a submodel 710, and the submodel 710D includes a plurality of convolutional layers 720D (also referred to as 770) as a plurality of convolutional layers 720.

[0421] In each convolutional layer 720D (770) within the submodel 710 in Figure 42, each data sequence is displayed three-dimensionally, similar to Figure 38. The total of six data sequences (three phase data sequences and three norm data sequences) arranged in the depth direction (diagonal direction) at the beginning of each convolutional layer 720 in Figure 42 are two-dimensionally arranged data sequences (two-dimensional array data), and their number of channels Nc is "1".

[0422] The learning model 410Q includes a convolutional layer 770 for each of the N data sequences that performs a convolution operation (two-dimensional convolution) on two or more (e.g., three) phase data sequences and two or more (e.g., three) norm data sequences at each spatial position Cn. For example, the two or more phase data sequences at each spatial position Cn are generated by duplicating a data sequence that shows the time change Φn(τ) of the phase of the complex signal at each spatial position Cn. Similarly, the two or more norm data sequences at each spatial position Cn are generated by duplicating a data sequence that shows the time change of the norm of the complex signal at each spatial position Cn. Here, three norm data sequences are generated by duplicating one data sequence that shows the time change Ψn(τ) of the norm of the complex signal at each spatial position Cn three times. However, it is not limited to this, and the three norm data sequences may include the original norm data sequence itself. The same applies to the phase data sequences.

[0423] In each convolutional layer 770 of the learning model 410Q, a convolution process is performed on the data sequence for each spatial position. In each convolutional layer 770, as in the ninth embodiment, both a data sequence showing the time change Φn(τ) of the phase of the complex signal and a data sequence showing the time change Ψn(τ) of the norm are used for each spatial position.

[0424] In each convolutional layer 770 of the learning model 410Q, a two-dimensional convolution process is performed, similar to the convolutional layer 630C of the learning model 410K (Figure 38).

[0425] However, in each convolutional layer 770, the phase data sequence (a data sequence showing the time change of the phase Φn(τ)) and the norm data sequence (a data sequence showing the time change of the norm Ψn(τ)) are each duplicated three times at each spatial position, generating a total of six data sequences as a two-dimensional data sequence. Subsequently, a convolution process similar to the convolution process in each convolutional layer 630C (Figure 38) (see the second modified example in the eighth embodiment) is performed on the two-dimensional data sequence, and output data from each convolutional layer 770 is obtained. As a result, for each of the N data sequences, a data sequence (a data sequence of size 1 × 240 × 1) (a 240-dimensional vector) is generated in which the characteristics of the time change of the phase according to the norm are extracted.

[0426] In this way, each of the N convolutional layers 770 performs a convolution operation on two or more identical data sequences at each spatial position, generating each data sequence after feature extraction. Then, the N convolutional layers 770 generate N data sequences after feature extraction. This makes it possible to obtain N data sequences (data sequences after feature extraction) in which the phase strength is well adjusted according to the magnitude of the norm of the complex signal.

[0427] Then, the N output data (N data sequences) from the N convolutional layers 770 are used as multiple channel data input to the subsequent submodel 810.

[0428] A learning model 410Q of this nature may be used.

[0429] In this third modified example, the convolution process is performed on two adjacent columns selected from the six data columns of the first stage (a total of six data columns consisting of three phase data columns and three norm data columns), while repeatedly selecting two adjacent columns. This generates a five-column data column (arranged two-dimensionally) (a data column of size 5 × 240) for the next stage (second stage). Furthermore, the same process is repeated to sequentially generate the data columns for the third stage and beyond.

[0430] However, the present invention is not limited thereto. For example, first, a convolution process may be repeated (twice) on two adjacent columns of three phase data columns to generate a representative phase data column, and then a convolution process may be repeated (twice) on two adjacent columns of three norm data columns to generate a representative norm data column. Then, a convolution process may be performed on the representative phase data column and the representative norm data column to generate output data from the convolutional layer 770. A total of N output data (N data columns) from N convolutional layers 770 may be used as multiple (N) channel data input to a subsequent submodel 810.

[0431] <Further Modifications of the Eighth and Ninth Embodiments> In the eighth and ninth embodiments, etc. (including their respective modifications), time scale conversion, channel shuffling (Figure 18), etc., may be performed, similar to the first embodiment. For example, the channel shuffling in the eighth embodiment may be performed at the input stage to the subsequent submodel 810, or at the input stage to the preceding submodel 610.

[0432] Furthermore, in the eighth embodiment and the like, a layer for convolving multiple data sequences showing the time change of the phase of a complex signal at the same spatial position is provided individually for each data sequence. Specifically, a convolution layer 630 that performs convolution processing on data sequences showing the time change of the phase of a complex signal at the same spatial position is provided individually for each of the N data sequences. In other words, the parameter values ​​of each convolution layer 630 (for each data sequence) are different for each data sequence. However, this is not limited to this. For example, the convolution layer 630 may be provided in common for all N data sequences. In other words, the same convolution layer 630 (with the same parameter values) may be used to perform convolution processing on each of the N data sequences.

[0433] Thus, the layer (630) that convolves multiple data sequences representing the time evolution of the phase of a complex signal at the same spatial position may be provided individually for each of the multiple data sequences, or it may be provided in common for all of the multiple data sequences. In other words, the convolution process performed for each of the multiple data sequences may be performed using a convolution layer common to all of the multiple data sequences, or it may be performed using individual convolution layers provided separately for each of the multiple data sequences.

[0434] Similarly, in the ninth embodiment and the like, the convolutional layer 720 may be provided in common for multiple data sequences. The same convolutional layer 720 (with the same parameter values) may be used to perform convolution for each of the multiple data sequences. Thus, the layer (720) that convolves multiple data sequences at the same spatial position may be provided individually for each of the multiple data sequences, or it may be provided in common for multiple data sequences.

[0435] Furthermore, while an hourglass-shaped network model is exemplified as the learning model 410 (deep neural network model) in the eighth and ninth embodiments described above, the invention is not limited thereto. For example, a diffusion model may be used as the learning model 410 (deep neural network model). More specifically, a plurality of data sequences (639D (Figure 36), etc.) after feature extraction by convolution processing may be used as the input data (N input data) 351D input to the encoder 352 (encoder used to generate the condition vector 353) of the diffusion model (see Figure 21). In other words, a new plurality of data sequences (639D (Figure 36), etc.) after feature extraction, generated by convolution processing by convolution layers 630 or 720 (see Figures 36 to 42) for each of the original plurality of data sequences based on signals of each spatial position, may be used as the input data (N input data) 351D to the encoder 352.

[0436] Furthermore, while the eighth and ninth embodiments described above illustrate embodiments that do not involve the estimation of the optimal position, the model is not limited thereto. For example, the eighth and ninth embodiments may also include the estimation of the optimal position, similar to the seventh embodiment.

[0437] <10. Others> Although embodiments of the present invention have been described above, the present invention is not limited to those described above.

[0438] In the embodiments described above, the time change Φ(τ) of the phase Φ of the complex signal with respect to each spatial position Cn(r, θ) is exemplified as the data sequence for the reflected wave, but the embodiments are not limited to this. For example, a data sequence showing the time change of the "norm" (signal intensity) of the complex signal with respect to each spatial position Cn(r, θ) may be used as the data sequence.

[0439] Specifically, in the first to third embodiments and the fifth to seventh embodiments, etc., the N data sequences may be data sequences that show the time evolution of the "norm" of N complex signals relating to N combinations Cn(r, θ) (N spatial positions). Furthermore, in the fourth embodiment, etc., a high-pass filter, an absolute value filter, and a median filter may be applied to a data sequence that shows the time evolution of the "norm" of a complex signal relating to a certain combination Cn(r, θ) to generate a heart rate cycle waveform.

[0440] Furthermore, while the above embodiments use data sequences based on the signal (complex signal) after static clutter has been removed, the invention is not limited to this. For example, in the above embodiments, data sequences showing the time change of the complex signal before static clutter is removed (time change of phase Φ Φn(τ) (n=1,...,N)) may be used as N data sequences. Also, when simply using the "change" of the signal, such as when static clutter removal is not performed, the virtual origin (see Figure 7) may be placed anywhere.

[0441] Furthermore, in the embodiments described above, the N data sequences are exemplified by the complex signal (phase Φ, etc.) after applying a high-pass filter, but the embodiments are not limited to this. For example, in the embodiments described above, the N data sequences may be a data sequence showing the time change of the complex signal before applying a high-pass filter (time change of phase Φ, etc.).

[0442] Furthermore, the high-pass filter in each embodiment may, but is not limited to, directly attenuating the low-frequency components from the original signal. For example, the high-pass filter may be a filter (signal processor) that first generates a signal with the high-frequency components of the original signal attenuated (high-frequency attenuated signal), and then subtracts this high-frequency attenuated signal (the signal with the high-frequency components attenuated) from the original signal, thereby selectively passing high-frequency components.

[0443] Furthermore, while complex signals (such as phase Φ) are exemplified as signals at each spatial position in the above embodiments, the invention is not limited to these. For example, the signal at each spatial position may consist only of components corresponding to the real part (real number portion) of the complex signal. In other words, the signal at each spatial position may be a signal that does not have orthogonal components of the complex signal. A data sequence or the like showing the time change Ψ(τ) of the norm Ψ (signal intensity) of the signal may be used. In this case, static clutter removal may be performed, for example, by subtracting the average value of the signal (multiple digital values) over a certain period as the static clutter component (DC component) from the original signal (original value).

[0444] Furthermore, in the embodiments described above, only one angle θ in one direction is considered as the direction of arrival (arrival direction) of the reflected wave, but this is not limited to this. Two angles (θ, ψ) in two different directions may be considered as the direction of arrival (arrival direction). More specifically, the millimeter-wave sensor 20 may acquire complex signals relating to multiple spatial positions expressed in spherical coordinates (r, θ, ψ) in three-dimensional space. Then, the heart rate cycle waveform and / or optimal position may be estimated based on a data sequence based on these complex signals relating to multiple spatial positions.

[0445] Furthermore, the number of channels in the input data of the learning model 410 in each of the above embodiments (the number of multiple channels in the input data) does not have to be the same as the number of multiple data sequences measured by the millimeter-wave sensor 20.

[0446] Specifically, the number of channels may be a value greater than the number of data sequences (N sequences) (N + β) (where β is a natural number). For example, some of the data sequences (β data sequences) from the N data sequences relating to N spatial positions Cn(r, θ) may be overlapped to generate a total of (N + β) input data channels.

[0447] Alternatively, conversely, the number of channels may be a value (N - β) smaller than the total number (N) of a plurality of data sequences measured by the millimeter-wave sensor 20. For example, a part of the N data sequences ((N - β) data sequences) measured for N spatial positions Cn(r, θ) may be selected and arranged in a total of (N - β) channels. Further, when (N - β) data sequences are appropriately selected from the N data sequences as input data of a plurality of teacher data, the (N - β) data sequences to be selected may be different for each teacher data.

[0448] In the sixth embodiment, the seventh embodiment, etc., when the number of channels of the input data 511D is a value (N + β) larger than the number (N) of a plurality of data sequences, it is preferable that the data sequence at the optimal position is not duplicated within the input data 511D of the teacher data so that only one data sequence at the optimal position is adopted. In the sixth embodiment, the seventh embodiment, etc., when the number of channels of the input data 511D is a value (N - β) smaller than the number (N) of a plurality of data sequences, it is preferable that only one data sequence at the optimal position is selected and arranged within the input data 511D of the teacher data. Alternatively, the position corresponding to the best data (also referred to as the second-best position) among the arbitrarily selected (N - β) spatial positions from the N positions may be set as the optimal position, and the data sequence at the second-best position may be included in the input data 511D of the teacher data, and the second-best position may be given as the correct answer data 518.

[0449] The measurement target space and the measurement target person are not limited to the space in front of the ATM and the operator of the ATM, and may be various other places and various people (for example, the space in front of the cash register in a store (settlement area) and the person to be settled, etc.).

[0450] In each of the above embodiments, etc., various processes (data acquisition process, generation process of the learning model 410, and inference process using the learning model 410 (learned model 420), etc.) are executed by a single device 10, but it is not limited thereto. For example, these various processes may be shared and executed by a plurality of devices.

[0451] 10 Heart rate measurement device 20 Millimeter wave sensor 21 Transmitter 22 Transmitting antenna 23 Receiving antenna 24a, 24b Mixer 110 Determined area (measurement target area) 310, 330 Hourglass network model 350, 370 Spread spectrum model 410 Learning model Bi Period Cn Spatial position SI Common phase signal SQ Quadrature signal

Claims

1. A heart rate measuring device comprising: a control means for estimating the heart rate cycle waveform of a single living organism by acquiring a signal relating to the reflected wave of the transmitted wave and a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor, and inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the heart rate cycle waveform when the plurality of data sequences based on the plurality of signals are input.

2. The heart rate measuring device according to claim 1, wherein the signal relating to the reflected wave is a complex signal composed of an in-phase signal and an orthogonal signal of the reflected wave, and the data sequence input to the learning model is a data sequence showing the time change of the phase of the complex signal.

3. The heart rate measurement device according to claim 2, wherein, in the learning model, a convolution process is performed on each of the plurality of data sequences with respect to two or more data sequences that show the time change of the phase of the complex signal at the same spatial position.

4. The heart rate measurement device according to claim 2, wherein, in the learning model, a convolution process is performed on each of the plurality of data sequences to generate two or more data sequences that show the time change of the phase of the complex signal at the same spatial position, thereby generating two or more data sequences.

5. The heart rate measuring device according to claim 1, wherein the signal relating to the reflected wave is a complex signal composed of an in-phase signal and an orthogonal signal of the reflected wave, the data sequence input to the learning model has a data sequence showing the time change of the phase of the complex signal and a data sequence showing the time change of the norm of the complex signal, and in the learning model, a convolution operation is performed on each of the plurality of data sequences with respect to the data sequence showing the time change of the phase of the complex signal at the same spatial location and the data sequence showing the time change of the norm of the complex signal at the same spatial location.

6. The heart rate measuring device according to claim 5, wherein in the convolution process for each of the plurality of data sequences, a convolution process is performed on two or more data sequences that show the time change of the phase of the complex signal at the same spatial position to generate a convolutional phase data sequence which is a data sequence of two or more channels; a convolution process is performed on two or more data sequences that show the time change of the norm of the complex signal at the same spatial position to generate a convolutional norm data sequence which is a data sequence of two or more channels; and a convolution process is also performed on a data sequence of four or more channels which is an integrated convolutional phase data sequence and a convolutional norm data sequence.

7. The heart rate measurement device according to any one of claims 3 to 6, wherein the learning model is a deep neural network model and comprises an encoder with downsampling and a decoder with upsampling, and a new plurality of data sequences generated by feature extraction through a convolution process on each of the plurality of data sequences are used as a plurality of channel data input to the encoder.

8. The heart rate measurement device according to claim 1 or 2, wherein the data sequence input to the learning model includes a data sequence showing a provisional heart rate period waveform, which is a provisional heart rate period waveform based on an oscillatory wave mainly containing harmonic components of the heart rate, extracted from the signal.

9. The heart rate measurement device according to claim 1 or 2, wherein the learning model is a deep neural network model comprising an encoder with downsampling and a decoder with upsampling.

10. The heart rate measurement device according to claim 1 or 2, wherein the learning model is a deep neural network model, and the deep neural network model is a diffusion model.

11. The heart rate measuring device according to any one of claims 1 to 10, wherein the plurality of spatial positions indicated by the plurality of combinations of angle and distance are positions within a predetermined region that is set in advance as a region in which a part of the one living organism may be included.

12. The heart rate measuring device according to any one of claims 1 to 10, wherein the plurality of spatial positions indicated by the plurality of combinations of angle and distance include positions having a signal intensity of a predetermined degree or higher.

13. The heart rate measuring device according to any one of claims 1 to 12, wherein the control means detects a plurality of peaks in the heart rate cycle waveform estimated by the learning model and determines the heart rate based on the plurality of peaks.

14. The heart rate measurement device according to any one of claims 1 to 12, wherein each of the plurality of data sequences based on the plurality of signals is data for a predetermined period length, the learning model is a model that has been trained to output the heart rate cycle waveform when the plurality of data sequences over the predetermined period length are input, and the control means inputs the plurality of data sequences over the predetermined period length to the learning model for a plurality of different periods, each having the predetermined period length, by sliding the start time of the period by a predetermined minute 15. The heart rate measurement device according to any one of claims 1 to 12, wherein each of the plurality of data sequences based on the plurality of signals is data for a predetermined period length, the learning model is a model that has been trained to output the heart rate cycle waveform when the plurality of data sequences over the predetermined period length are input, and the control means inputs the plurality of data sequences over the predetermined period length to the learning model for a plurality of different periods, each having the predetermined period length, by sliding the start time of the period by a predetermined minute by a predetermined minute, and obtains a provisional output waveform which is the heart rate cycle waveform output from the learning model, and estimates the peak position of the heart rate cycle waveform of the one living organism based on the peak position of each provisional output waveform obtained for each of the plurality of periods.

16. The heart rate measurement device according to any one of claims 1 to 15, wherein each input data during training of the learning model is data having a structure in which data sequences relating to each signal of the plurality of combinations are arranged in channels on a data sequence basis, and the learning model is a model trained using a plurality of input data which are obtained by changing the channel arrangement of the plurality of data sequences relating to the plurality of combinations, and the plurality of input data which include first input data in which the plurality of data sequences are arranged in a first arrangement in a plurality of channels, and second input data in which the plurality of data sequences are arranged in a second arrangement in a plurality of channels.

17. A heart rate measurement method comprising: a) acquiring a signal relating to the reflected wave of a transmitted wave and acquiring a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor; and b) estimating the heart rate cycle waveform of the single living organism by inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a model that has been machine-trained to output the heart rate cycle waveform when the plurality of data sequences based on the plurality of signals are input.

18. A program for causing a computer to perform the following steps: a) acquire a signal relating to the reflected wave of a transmitted wave and acquire a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor; and b) estimate the heart rate cycle waveform of the single living organism by inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the heart rate cycle waveform when it is input to a plurality of data sequences based on the plurality of signals.

19. A learning model generation device comprising: a control means for machine learning a learning model that takes as input a plurality of data sequences relating to a plurality of spatial locations near a single living organism, which are indicated by a plurality of combinations of the distance between the control means and a sensor transmitting a transmission wave and an angle indicating the direction from the sensor, and outputs a heart rate cycle waveform, wherein each of the plurality of data sequences is a data sequence based on a signal relating to the reflected wave of the transmission wave, and the control means machine learning the learning model based on training data comprising the plurality of data sequences and the correct data of the heart rate cycle waveform.

20. The learning model generation device according to claim 19, wherein the control means performs machine learning on the learning model not only on first training data comprising the plurality of data sequences and the correct data of the heart rate cycle waveform, but also on second training data comprising modified plurality of data sequences obtained by changing the time scale of the plurality of data sequences by a predetermined ratio, and modified correct data obtained by changing the time scale of the correct data by the predetermined ratio.

21. The learning model generation apparatus according to claim 19, wherein each input data during the training of the learning model is data having a structure in which data sequences relating to each signal of the plurality of combinations are arranged in channels on a data sequence basis, and the control means performs machine learning on the learning model using a plurality of input data which are obtained by changing the channel arrangement of the plurality of data sequences relating to the plurality of combinations, the plurality of input data which include first input data in which the plurality of data sequences are arranged in a first arrangement in a plurality of channels, and second input data in which the plurality of data sequences are arranged in a second arrangement in a plurality of channels.

22. The learning model generation device according to claim 19, wherein each input data during the training of the learning model is data having a structure in which data sequences relating to each of the multiple combinations of signals are arranged in channels on a data sequence basis, and the control means uses data in which some of the multiple data sequences relating to the multiple combinations are replaced with zeros as training data to machine-learn the learning model.

23. A trained model for causing a computer to output a heart rate cycle waveform when it is input a plurality of data sequences relating to a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a wave and an angle indicating the direction from the sensor, comprising: each of the plurality of data sequences is a data sequence relating to the reflected wave of the transmitted wave; and the trained model is a model that has been machine-learned based on training data comprising the plurality of data sequences for training and ground truth data of the heart rate cycle waveform.

24. A heart rate measuring device comprising: a control means for estimating the optimal position among the multiple spatial positions from which the best data relating to the heart rate of the single living organism can be obtained, by acquiring a signal relating to the reflected wave of the transmitted wave and a data sequence based on the signal for each of the multiple spatial positions near a single living organism, which are represented by a plurality of combinations of the distance between the sensor transmitting the transmitted wave and the angle indicating the direction from the sensor, and inputting the plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the optimal position when the plurality of data sequences based on the plurality of signals is input.

25. A heart rate measurement method comprising: a) acquiring a signal relating to the reflected wave of a transmitted wave and acquiring a data sequence based on the signal for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor; and b) estimating the optimal location among the plurality of spatial locations from which the best data relating to the heart rate of the single living organism can be obtained by inputting the plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model, wherein the learning model is a machine learning model that outputs the optimal location when the plurality of data sequences based on the plurality of signals is input.

26. A program for causing a computer to perform the following steps: a) for each of a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor that transmits a transmitted wave and an angle indicating the direction from the sensor, a step of acquiring a signal relating to the reflected wave of the transmitted wave and acquiring a data sequence based on the signal; b) for inputting a plurality of data sequences based on the plurality of signals relating to the plurality of combinations into a learning model to estimate the optimal location among the plurality of spatial locations from which the best data relating to the heart rate of the single living organism can be acquired, wherein the learning model is a machine learning model that outputs the optimal location when the plurality of data sequences based on the plurality of signals is input.

27. A learning model generation device comprising: a control means for machine learning a learning model which takes as input a plurality of data sequences relating to a plurality of spatial positions near a single living organism, which are indicated by a plurality of combinations of the distance between the spatial position and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor, and outputs the optimal position among the plurality of spatial positions from which the best data relating to the heartbeat of the single living organism can be obtained, wherein each of the plurality of data sequences is a data sequence based on a signal relating to the reflected wave of the transmitted wave, and the control means machine learning the learning model based on training data comprising the plurality of data sequences and the correct data of the optimal position.

28. The learning model generation device according to claim 27, wherein the learning model is a model that has been trained to output a heart rate cycle waveform in addition to the optimal position, and the control means trains the learning model based on training data comprising the plurality of data sequences, the correct data of the optimal position, and the correct data of the heart rate cycle waveform.

29. A trained model for causing a computer to function by taking a plurality of data sequences relating to a plurality of spatial locations near a single living organism, which are represented by a plurality of combinations of the distance between the spatial location and a sensor transmitting a transmitted wave and an angle indicating the direction from the sensor as input, and outputting the optimal location among the plurality of spatial locations from which the best data relating to the heartbeat of the single living organism can be obtained, wherein each of the plurality of data sequences is a data sequence based on a signal relating to the reflected wave of the transmitted wave, and the trained model is a model that has been machine-learned based on training data comprising the plurality of data sequences for training and the ground truth data of the optimal location.

Images