Object characteristic estimation device, method, and program

Abstract

In one aspect of the present invention: first measurement data representing an acupressure force when an acupressure operation is performed on an object, and second measurement data including information representing the three-dimensional position of the acupressure operation on the object are acquired; three-dimensional position information representing an acupressure position on the object is obtained from the acquired second measurement data; and a displacement amount of the object by the acupressure operation is calculated on the basis of the calculated three-dimensional position information. Then, feature amount information representing the elasticity and viscosity of the object is acquired on the basis of the acupressure force represented by the first measurement data and the calculated displacement amount; and the state of tissue of the object is estimated on the basis of the acquired feature amount information representing the elasticity and viscosity.

Description

Apparatus, method, and program for estimating object properties 【0001】 One aspect of this invention relates to an object property estimation apparatus, method, and program used, for example, to measure the tactile sensation of human skin tissue. 【0002】 In recent years, the worsening shortage of doctors and the increasing depopulation of rural areas have made it increasingly difficult to sustain medical care that relies on face-to-face consultations. As a result, various telemedicine systems utilizing information and communication technology (ICT) are being developed, one of which is a system that allows for remote palpation of the patient's affected area, including its hardness, texture, and shape. 【0003】 For example, Non-Patent Document 1 describes a method for acquiring palpation information of an affected area using a robotic hand equipped with pressure and temperature sensors. Non-Patent Document 2 proposes a method in which a caregiver or family member, for example, attaches tactile sensors to their fingers or other body parts, touches the affected area to measure tactile information, and transmits the measured tactile information to a doctor's terminal via a communication network, allowing the doctor to perform a remote palpation. 【0004】 Shingo Shimoda, "Human Intelligence and Palpation - Elucidating the Cognitive Pathways of Palpation and Considering the Realization of Remote Palpation," Journal of the Robotics Society of Japan, Vol. 40, No. 8, pp. 659-664, October 2022, Internet <URL: https: / / www.jstage.jst.go.jp / article / jrsj / 40 / 8 / 40_40_659 / _pdf> Hokkaido University, "Successful 5G Remote Palpation Experiment Connecting Core Hospitals in Hokkaido," PLESS RELEASE 2024 / 1 / 10, Internet <URL: https: / / www.hokudai.ac.jp / news / pdf / 240110_pr2.pdf> 【0005】 However, the method described in Non-Patent Document 1 can only measure the elastic modulus of the affected area. Edema is one of the conditions treated by palpation, but since edema involves changes in both the elasticity and viscosity of the tissue, measuring only the elastic modulus does not allow for determination of whether the skin is simply soft or if there is edema, and if it is edema, it is difficult to determine its severity. 【0006】On the other hand, while the method described in Non-Patent Document 2 states that it can measure both the elasticity and viscosity of tissue, it does not describe the measurement method or the method for determining the state of the tissue based on the measurement results. 【0007】 This invention was made in view of the above circumstances, and aims to provide a technology that enables accurate estimation of the structural state of an object by establishing a method for measuring the viscoelasticity of the object. 【0008】 To solve the above problems, one embodiment of the object property estimation device or estimation method according to the present invention comprises: a first processing unit that acquires first measurement data representing the finger pressure when a finger pressure operation is performed on an object, and second measurement data including information representing the three-dimensional position of the finger pressure operation on the object; a second processing unit that obtains three-dimensional position information representing the finger pressure position on the object from the second measurement data and calculates the amount of displacement of the object due to the finger pressure operation based on the calculated three-dimensional position information; a third processing unit that acquires feature quantity information representing the elasticity and viscosity of the object based on the finger pressure and the amount of displacement represented by the first measurement data; and a fourth processing unit that estimates the state of the structure of the object based on the feature quantity information representing the elasticity and viscosity. 【0009】 According to one aspect of this invention, a method for measuring the viscoelasticity of an object is established, thereby providing a technology that enables accurate estimation of the structural state of the object. 【0010】Figure 1 is a diagram showing an example of the overall configuration of a remote palpation system using an object property estimation device according to one embodiment of the present invention. Figure 2 is a block diagram showing an example of the hardware configuration of an object property estimation device according to one embodiment of the present invention. Figure 3 is a block diagram showing an example of the software configuration of an object property estimation device according to one embodiment of the present invention. Figure 4A is a flowchart used to explain the processing procedure and processing content of the palpation state estimation process executed by the control unit of the object property estimation device shown in Figure 3. Figure 4B is a flowchart used to explain the processing procedure and processing content of the palpation state estimation process executed by the control unit of the object property estimation device shown in Figure 3. Figure 4C is a flowchart used to explain the processing procedure and processing content of the palpation state estimation process executed by the control unit of the object property estimation device shown in Figure 3. Figure 5 is a diagram showing an example of information stored in the finger curvature radius storage unit of the object property estimation device shown in Figure 3. Figure 6 is a diagram showing an example of information stored in the finger pressure start point coordinate storage unit of the object property estimation device shown in Figure 3. Figure 7 is a diagram showing an example of information stored in the fingertip physical property value correction coefficient storage unit of the object property estimation device shown in Figure 3. Figure 8 is a diagram showing an example of information stored in the judgment model storage unit of the object property estimation device shown in Figure 3. Figure 9 is a diagram illustrating an example of palpation condition measurement operation for the affected area. Figure 10 is a diagram showing an example of the time change of finger pressure and displacement. Figure 11 is a diagram showing a magnified view of some of the characteristics shown in Figure 10. Figure 12 is a diagram showing an example of a confusion matrix showing the relationship between measured and predicted values ​​regarding the severity level of edema. Figure 13 is a diagram showing an example of a tactile presentation device. 【0011】 Embodiments of this invention will be described below with reference to the drawings. 【0012】 [One Embodiment] (Configuration Example) (1) System Figure 1 is a diagram showing an example of the overall configuration of a remote palpation system using an object property estimation device according to one embodiment of the present invention. 【0013】This remote palpation system involves placing an object property estimation device PS at a remote location, such as the patient's home or facility. The object property estimation device PS determines the tactile state of the affected area of ​​the patient's RP and the corresponding severity of the affected area based on video data from a depth camera CM and pressure measurement data from a pressure sensor SS. The system is configured to support the physician's diagnosis by transmitting information representing the determination result to a physician's terminal MT at the hospital via a network NW and presenting it to the physician. 【0014】 The pressure sensor SS is attached to the fingertip of a caregiver or family member (also called a user) who contacts the affected area of ​​the patient's RP on behalf of a physician. The attachment position is set on the fingernail so as not to interfere with the sensation of acupressure on the affected area. The pressure sensor SS acquires measurement data representing finger pressure by measuring the deformation of the fingernail when the caregiver applies acupressure to the affected area, and outputs this measurement data to an object property estimation device PS, for example, via a signal cable. 【0015】 The depth camera CM is positioned above the affected area and captures images of the fingertip pressure applied to the affected area by the caregiver. The resulting video data is output to the object property estimation device PS via a signal cable. The video data includes RGB image data and depth image data of the area being imaged. 【0016】 Furthermore, the pressure sensor SS and depth camera CM and the object property estimation device PS may be connected not only by a signal cable, but also by a wireless interface employing a low-power wireless data transmission standard such as Bluetooth® or Wi-Fi®. 【0017】 On the other hand, the physician terminal MT is, for example, composed of a personal computer and is used to display images representing the state of the finger pressure applied to the affected area, as well as information representing the tactile state of the affected area and the corresponding determination of the severity of the affected area, which are transmitted from the object property estimation device PS on the remote side. 【0018】Furthermore, a tactile sensation display device MS is connected to the physician terminal MT. The physician terminal MT reproduces the tactile sensation of the affected area by operating the tactile sensation display device MS based on information representing the result of the determination of the tactile sensation of the affected area. 【0019】 (2) Object property estimation device PS Figures 2 and 3 are block diagrams showing examples of the hardware configuration and software configuration of an object property estimation device PS according to one embodiment of the present invention. 【0020】 The object property estimation device PS is configured, for example, by a personal computer. Alternatively, the object property estimation device PS may be configured by a portable device such as a smartphone, tablet, head-mounted display (HMD), or smart glasses, or by a server computer located on the web or in the cloud. 【0021】 The object property estimation device PS includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU). The control unit 1 is connected to a storage medium having a program storage unit 2 and a data storage unit 3, an input / output interface (hereinafter referred to as I / F) unit 4, and a communication I / F unit 5 via a bus 6. 【0022】 The input / output interface 4 is connected to a depth camera CM and a pressure sensor SS, and further to an input device IN and a display device DP. Of these, the input device IN is used for operation command instructions to the object property estimation device PS and for the user to input their own finger size. The display device DP is used to show the user, who is acting as the agent, the appropriate finger pressure to apply to the affected area. 【0023】The program storage unit 2 is configured, for example, by combining a non-volatile memory that can be written to and read at any time, such as an SSD (Solid State Drive), and a non-volatile memory such as ROM (Read Only Memory), and stores application programs necessary to execute various controls according to one embodiment, in addition to middleware such as an OS (Operating System). Hereafter, the OS and each application program will be collectively referred to as a program. 【0024】 The data storage unit 3 combines, for example, a non-volatile memory such as an SSD that can be written to and read at any time as a storage medium, and a volatile memory such as RAM (Random Access Memory). In its storage area, as the main storage unit according to one embodiment of this invention, a finger curvature radius storage unit 31, a fingertip physical property value correction coefficient storage unit 32, a pressure point start point coordinate storage unit 33, and a judgment model storage unit 34 are provided. 【0025】 The finger curvature radius memory unit 31 is used to store information representing the size of the finger used by the substitute to perform acupressure on the affected area, for example, the finger curvature radius at the pad of the finger performing the acupressure. 【0026】 The fingertip physical property correction coefficient storage unit 32 stores correction coefficients for correcting the fingertip physical properties. More specifically, the elastic modulus (Young's modulus) of the user's finger when applying pressure to the affected area with the fingertip varies depending on the contact conditions with the affected area. For example, under low finger pressure, the pressure is mainly from the relatively soft epidermis, whereas under high finger pressure, the epidermis is compressed and the pressure is mainly from hard tissues such as bone. Therefore, the fingertip Young's modulus is higher under high finger pressure compared to low finger pressure. For this reason, a calibration experiment is performed in advance to calculate the correction coefficient for the fingertip Young's modulus with respect to finger pressure and displacement, and the calculated correction coefficient is stored in the fingertip physical property correction coefficient storage unit 32. 【0027】 Figure 7 shows an example of a correction coefficient stored in the fingertip physical property correction coefficient storage unit 32. In this example, the correction coefficient c is stored in correspondence with the finger pressure coefficient a and the displacement coefficient b. 【0028】The acupressure start point coordinate memory unit 33 is used to store the three-dimensional position coordinates (acupressure start point coordinates) at the moment when acupressure is started on the affected area. 【0029】 The determination model memory unit 34 stores parameters that are model data for a machine learning model that has been learned to output the severity of the affected area as an objective variable when the elastic modulus (Young's modulus) and viscous feature quantity of the affected area are input as explanatory variables. 【0030】 FIG. 8 shows an example of the parameters stored in the determination model memory unit 34. In this example, the weight coefficient sequence w １ , w ２ , …, w ｎ is stored. 【0031】 The control unit 1, as a processing function necessary to implement an embodiment of this invention, first includes, as a video processing system, a video data acquisition unit 11, a hand / sensor area determination unit 12, a background removal unit 13, a palpation finger three-dimensional coordinate calculation unit 14, a hand / sensor area removal unit 15, and a normal vector calculation unit 16. 【0032】 The control unit 1 also includes, as an acupressure force processing system, a finger curvature radius acquisition unit 17, a pressure data acquisition unit 18, and a fingertip pressure presentation unit 19. 【0033】 Further, the control unit 1 includes, as an arithmetic processing system, a displacement amount calculation unit 21, a Young's modulus calculation unit 22, a viscous feature quantity calculation unit 23, and a severity determination unit 24. 【0034】 Each of the above processing units 11 to 16, 17 to 19, 21 to 24 is realized by causing the hardware processor of the control unit 1 to execute an application program stored in the program memory unit 2. Note that a part or all of the above processing units 11 to 16, 17 to 19, 21 to 24 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit). 【0035】The video data acquisition unit 11 acquires the video data output from the depth camera CM via the input / output I / F unit 4, and temporarily stores the acquired video data in a video storage area (not shown) in the data storage unit 3. 【0036】 The hand / sensor area determination unit 12 extracts, from the RGB image data included in the acquired video data, the area including the hand of the surrogate and the pressure sensor SS. 【0037】 The background removal unit 13 performs filtering processing on the depth image data included in the video data using a preset threshold value, thereby removing the background area other than the hand of the surrogate and the pressure sensor SS from the depth image data. 【0038】 The palpation finger three-dimensional coordinate calculation unit 14 first invalidates the background image components included in the RGB image area including the hand of the surrogate and the pressure sensor SS extracted by the hand / sensor area determination unit 12, using the depth image data from which the background area has been removed by the background removal unit 13. Then, the palpation finger three-dimensional coordinate calculation unit 14 calculates the three-dimensional position coordinates representing the central position of the pressure sensor SS, that is, the palpation finger (also referred to as the finger pressure finger), based on the RGB image data representing the area including the hand and the pressure sensor SS from which the background area has been invalidated. An example of the calculation process of the three-dimensional position coordinates will be described in detail in the operation example. 【0039】 The hand / sensor area removal unit 15 generates depth image data from which the area including the hand and the pressure sensor SS has been removed, based on the RGB image data of the area including the hand and the pressure sensor SS extracted by the hand / sensor area determination unit 12, from the depth image data from which the background area has been removed by the background removal unit 13. 【0040】 The normal vector calculation unit 16 calculates a three-dimensional normal vector perpendicular to the surface of the palpation target site, based on the depth image data from which the area including the hand and the pressure sensor SS has been removed. An example of the calculation process of the three-dimensional normal vector will be described in the operation example. 【0041】The finger curvature radius acquisition unit 17 acquires the curvature radius of the pad of the finger used for the acupressure action, which is input by the substitute via the input device IN prior to measurement, via the input / output I / F unit 4, and stores the acquired finger curvature radius in the finger curvature radius storage unit 31. 【0042】 The pressure data acquisition unit 18 acquires the finger pressure measurement data output from the pressure sensor SS when the substitute applies pressure to the affected area via the input / output I / F unit 4, and temporarily stores it in the storage area of ​​the data storage unit 3. 【0043】 The fingertip pressure display unit 19 outputs the acquired finger pressure measurement data, along with data representing the appropriate target finger pressure, from the input / output I / F unit 4 to the display device DP, and displays it for the agent to see. 【0044】 The displacement calculation unit 21 calculates the displacement along the three-dimensional normal vector based on the finger pressure measurement data acquired by the pressure data acquisition unit 18 and the three-dimensional coordinates of the central position of the palpation target area calculated by the palpation finger three-dimensional coordinate calculation unit 14. 【0045】 The Young's modulus calculation unit 22 calculates the elastic modulus (Young's modulus) of the affected area based on the measurement data of the finger pressure acquired by the pressure data acquisition unit 18 and the change in the amount of displacement calculated by the displacement amount calculation unit 21 during the finger pressure massage. An example of this Young's modulus calculation process will be explained in detail in the operation example. 【0046】 The viscosity feature calculation unit 23 calculates the viscosity feature of the edema based on the measurement data of the finger pressure acquired by the pressure data acquisition unit 18 and the change in the amount of displacement calculated by the amount of displacement calculation unit 21. 【0047】 The severity determination unit 24 uses a machine learning model whose parameters are stored in the determination model storage unit 34 to determine the severity of edema, which represents the state of the tissue of the object, based on the Young's modulus calculated by the Young's modulus calculation unit 22 and the viscosity feature quantity calculated by the viscosity feature quantity calculation unit 23. 【0048】 (Example of operation) Next, an example of the operation of the object property estimation device PS configured as described above will be explained. 【0049】Figures 4A to 4C are flowcharts showing an example of the processing procedure and content of the palpation state estimation process performed by the control unit 1 of the object property estimation device PS. 【0050】 (1) Prior to performing the finger curvature radius acquisition, the person acting as the agent inputs the curvature radius of their own finger to be used for the acupressure action via the input device IN. In response, the control unit 1 of the object property estimation device PS, under the control of the finger curvature radius acquisition unit 17, acquires the input curvature radius of the palpated finger via the input / output I / F unit 4 in step S11, and stores the acquired curvature radius of the palpated finger in the finger curvature radius storage unit 31. 【0051】 Figure 5 shows an example of the radius of curvature of a palpation finger stored in the finger curvature radius memory unit 31. In this example, 15 mm is stored as the finger curvature radius. 【0052】 The finger curvature radius may be estimated and stored, for example, by applying image recognition technology to the image of the finger captured by a depth camera (CM). 【0053】 (2) When the person providing the appropriate finger pressure begins applying finger pressure to the affected area, the finger pressure is measured by the pressure sensor SS and the measurement data is output. 【0054】 In response, the control unit 1 of the object property estimation device PS, under the control of the pressure data acquisition unit 18, first acquires the measurement data via the input / output interface unit 4 in step S12. Subsequently, in step S13, the control unit 1 of the object property estimation device PS, under the control of the fingertip pressure presentation unit 19, reads data representing an appropriate target finger pressure that has been previously stored from the storage area of ​​the data storage unit 3. The fingertip pressure presentation unit 19 then outputs the read appropriate reference finger pressure data, along with the acquired measurement data of the substitute's finger pressure, to the display device DP via the input / output interface unit 4 for display. 【0055】 The above-mentioned standard finger pressure data can be, for example, text data, graph data, graphic data, or animation, but is not limited to these and can be selected arbitrarily. 【0056】Generally, family members, caregivers, and other substitutes often lack knowledge of the appropriate finger pressure to apply during palpation. Therefore, by presenting the substitute with an appropriate target finger pressure, in conjunction with their finger pressure movements, as described above, the substitute can apply the correct pressure to the affected area. This function is extremely important for determining the severity of edema. 【0057】 The appropriate target pressure is set in advance, for example, based on the case. However, the diagnosing physician may visually inspect the affected area based on video data representing its condition transmitted from the object property estimation device PS, and then transmit the appropriate target pressure from the physician's terminal MT to the object property estimation device PS to set it. In addition, voice messages may be used as a method for presenting the target pressure, in addition to display data. 【0058】 (3) Calculation of three-dimensional position coordinates and normal vectors of palpation fingers using video data (3-1) Acquisition of video data When the acupressure motion is started, the motion is captured by the depth camera CM and the video data is output. In response, the control unit 1 of the object property estimation device PS acquires the video data output from the depth camera CM via the input / output I / F unit 4 in step S14 under the control of the video data acquisition unit 11 and temporarily stores the acquired video data in the video storage area of ​​the data storage unit 3. 【0059】 The video data acquisition unit 11 may also generate three-dimensional point cloud data from the video data and store this three-dimensional point cloud data. 【0060】 (3-2) Calculation of three-dimensional position coordinates of the palpation finger The control unit 1 of the object characteristic estimation device PS first extracts the region containing the substitute's hand and pressure sensor SS from the RGB image data included in the video data under the control of the hand / sensor region determination unit 12 in step S15. 【0061】 For example, the hand / sensor area determination unit 12 uses the YOLOv8 segmentation model, which is one of the object recognition AI models, to extract the area containing the surrogate's hand and pressure sensor SS from the RGB image data. 【0062】In addition, in step S16, the control unit 1 of the object characteristic estimation device PS performs filtering on the depth image data included in the video data using a preset threshold, under the control of the background removal unit 13, thereby removing background areas other than the surrogate's hand and the pressure sensor SS from the depth image data. The threshold is set, for example, to exclude objects located at a certain distance or more from the depth camera CM. 【0063】 In step S17, the control unit 1 of the object property estimation device PS, under the control of the palpation finger three-dimensional coordinate calculation unit 14, first invalidates the background image included in the RGB image region containing the substitute's hand and pressure sensor SS extracted by the hand / sensor region determination unit 12 by referring to the depth image data from which the background region has been removed by the background removal unit 13. 【0064】 Then, the palpation finger three-dimensional coordinate calculation unit 14 extracts the region containing only the pressure sensor SS, i.e., the palpation finger, based on the RGB image data representing the region containing the hand and pressure sensor SS, from which the background region has been disabled, and calculates the central position of the extracted region of the palpation finger as the palpation finger three-dimensional position coordinate. 【0065】 Figure 9 illustrates an example of the hand / sensor area determination process and the calculation process for the three-dimensional coordinates of the palpation finger described above. In the figure, E1 shows the extraction result of the image area that includes the substitute's hand FG and pressure sensor SS. Also, in the figure, E2 shows the area for calculating the average coordinate, calculated using the average value of the three-dimensional position coordinates of the pixels of the palpation finger / hand among the n×n pixels in the area surrounding the palpation finger. In the figure, O shows the three-dimensional position coordinate of the palpation finger, which is the center point of this area. 【0066】 Furthermore, the palpation finger coordinates in the RGB image may be obtained using, for example, an AR (Augmented Reality) marker, an infrared reflective marker, or a hand skeleton recognition model. In addition, the palpation finger's three-dimensional position coordinates may be smoothed using a moving average or similar method as appropriate. Moreover, to address outliers, the palpation finger's three-dimensional position coordinates may be calculated using the average value of the three-dimensional position coordinates of n x n pixels set around the palpation finger. 【0067】(3-3) Calculation of three-dimensional normal vectors The control unit 1 of the object characteristic estimation device PS then, under the control of the hand / sensor area removal unit 15, first in step S18, performs an expansion process on the hand / palpation finger area in the depth image data from which the background has been removed, and then in step S19, refers to the RGB image data of the area containing the hand and pressure sensor SS and removes the area containing the hand and pressure sensor SS from the depth image data. 【0068】 For example, the hand / sensor area removal unit 15 first specifies a pixel area n × m including the finger being palpated and its surroundings, and removes the area corresponding to the hand and pressure sensor SS determined by the hand / sensor area determination unit 12 from the specified pixel area n × m. Furthermore, based on the depth image data output from the background removal unit 13, it extracts only the depth image component of the object being palpated from which the background has been removed. 【0069】 Furthermore, in this process, pixels corresponding to the contour of the area recognized as the palpation finger or hand may remain in the extracted depth image, so the dilation process described above is performed. For example, a morphological transformation (dilation) process is applied to enlarge the recognition target area. Then, the depth image components of the palpation target are extracted from the depth image after the dilation process. 【0070】 In step S20, the control unit 1 of the object property estimation device PS calculates a three-dimensional normal vector perpendicular to the palpation target area based on the depth image data, which includes only the palpation target object, with the regions corresponding to the hand and pressure sensor SS removed, under the control of the normal vector calculation unit 16. 【0071】 For example, if the extracted depth information of the palpation target area consists of point cloud data, the normal vector calculation unit 16 performs appropriate decimation on this point cloud data and then calculates the normal vector using principal component analysis (PCA). When calculating the normal vector, it is preferable to perform smoothing on each pixel of the palpation target area using a moving average or the like. In addition to PCA, it is also possible to use a method other than PCA, such as meshing the point cloud and taking the average value of the normal vectors. 【0072】Thus, as shown in FIG. 9 for example, a three-dimensional normal vector PV that extends vertically upward with respect to the surface of the palpation target site from the three-dimensional position coordinates of the palpation finger is set. 【0073】 (4) Calculation of displacement amount Next, under the control of the displacement amount calculation unit 21, the control unit 1 of the object characteristic estimation device PS calculates the displacement amount of the affected part due to the finger pressure operation as follows. 【0074】 That is, the displacement amount calculation unit 21 first determines whether or not the measured value of the finger pressure is greater than "0" based on the measurement data of the finger pressure acquired by the pressure data acquisition unit 18 in step S21. As a result of this determination, if the measured value of the finger pressure is greater than "0", the displacement amount calculation unit 21 determines that the fingertip of the surrogate has contacted the affected part. 【0075】 Then, in step S22, the displacement amount calculation unit 21 determines whether or not the start point coordinates of the finger pressure operation have been stored in the finger pressure start point coordinate storage unit 33 of the data storage unit 3. If not stored, in step S23, the three-dimensional coordinates p ０ representing the start point of the finger pressure operation at the moment of contact are stored in the finger pressure start point coordinate storage unit 33. 【0076】 FIG. 6 shows an example of the finger pressure start point coordinates p ０ stored in the finger pressure start point coordinate storage unit 33. In this example, the case where x = 0.0012 m, y = 0.0021 m, and z = -0.0068 m are stored is shown. 【0077】 When the storage process of the three-dimensional coordinates p ０ of the finger pressure start point is completed, the control unit 1 of the object characteristic estimation device PS returns to step S12 and executes a series of processes from the acquisition of pressure data to the calculation of the normal vector by steps S12 to S20. As a result, the three-dimensional coordinates p ｉ of the finger pressure operation at time point i after the fingertip of the surrogate first contacts the affected part are obtained. 【0078】 On the other hand, in step S22, when it is determined that the start point coordinates p ０ of the finger pressure operation have been stored, the displacement amount calculation unit 21 reads the finger pressure start point coordinates p from the finger pressure start point coordinate storage unit 33 in step S24 ０ The data is read, and in step S25, the most recently calculated three-dimensional coordinates of the palpation finger p ｉ and the coordinates p of the starting point of the acupressure mentioned above. ０ The difference p ｉ -p ０ The displacement amount p is calculated. ｉ -p ０ The component along the normal vector n represents the displacement at time i. The displacement calculation unit 21 similarly calculates the displacement at each time i thereafter. Thus, data showing the change in the displacement of the affected area in response to the acupressure motion is obtained. 【0079】 (5) Calculation of Young's modulus The control unit 1 of the material property estimation device PS, under the control of the Young's modulus calculation unit 22, first reads the correction coefficient stored in the fingertip material property value correction coefficient storage unit 32 and the finger curvature radius stored in the finger curvature radius storage unit 31 at each of the above time points i in step S26. Next, in step S27, the Young's modulus calculation unit 22 calculates the Young's modulus using the measured value of finger pressure acquired by the pressure data acquisition unit 18, the displacement amount calculated by the displacement amount calculation unit 21, and the read correction coefficient and finger curvature radius, according to the calculation formula shown below. 【0080】 【0081】 Here, v ｉ ,v ｏ The Poisson ratio of the finger and the object being palpated, E ｉ , E ｏ The Young's modulus of the finger and the object being palpated is F. ｎ is the load, R ｉ δ is the radius of curvature of the finger. ｚｎ These indicate the amount of displacement. 【0082】 Note that the physical properties of the finger are (1-v ｉ ２ ) / E ｉ Since this value changes depending on the pressure applied during acupressure, the above reciprocal function is used as an approximation based on the results of calibration experiments using test pieces with known physical properties. 【0083】(6) Calculation of Viscosity Features In parallel with the Young's modulus calculation process described above, the control unit 1 of the object property estimation device PS, under the control of the viscosity feature calculation unit 23, calculates the viscosity features of the palpation target area in step S28 based on the measured value of finger pressure acquired by the pressure data acquisition unit 18 and the displacement amount calculated by the displacement amount calculation unit 21, as follows. 【0084】 In other words, the viscosity feature calculation unit 23 defines the viscosity feature of the edema using the residual displacement δres, the residual displacement ratio Rres, and the displacement change δdiff. Here, the residual displacement δres represents the displacement at the moment when the finger pressure f = 0 (non-contact), the residual displacement ratio Rres is the ratio of δres to the maximum displacement δmax, and the displacement change δdiff represents the difference in displacement between the start and end points of the finger pressure waveform (δend - δstart). 【0085】 Figure 10 shows an example of the measured finger pressure [N] and the corresponding change in the indentation amount [mm] of the palpation target area over time, while Figure 11 is an enlarged view of a portion of Figure 10. 【0086】 (7) Severity Determination Once the acupressure motion is completed, the control unit 1 of the object property estimation device PS, under the control of the severity determination unit 24, uses the calculation results of the Young's modulus and viscosity feature quantities to determine the severity of the palpation target area (affected area) as the state of the tissue of the object, as follows. 【0087】 In other words, the severity determination unit 24 first reads a sequence of weight coefficients, which are parameters of the machine learning model, from the determination model storage unit 34 in step S29, and then determines the severity corresponding to the calculation results of the Young's modulus and viscosity feature in step S30. 【0088】 Generally, edema that leaves an indentation after acupressure (pitting edema) is diagnosed based on how quickly the indentation returns after 10 seconds of pressure, and its severity is evaluated on a four-point scale from "1+" to "4+". Furthermore, the symptoms are classified as "Fast edema" or "Slow edema" depending on whether the indentation returns within 40 seconds or not. 【0089】Therefore, the severity determination unit 24 uses the elastic modulus (Young's modulus) and viscosity features as training data and trains a machine learning model to output a five-stage determination result, such as "normal" or "edema" (1+, ~, 4+), depending on multiple combinations of different values ​​for each of the above Young's modulus and viscosity features. Then, by inputting the calculated Young's modulus and viscosity features as explanatory variables into the trained machine learning model, the machine learning model obtains whether the patient is normal or edematous, and if edema, the severity level of that patient. 【0090】 Figure 12 shows an example of a confusion matrix illustrating the relationship between measured and predicted values ​​for edema severity levels when edema is assessed using LDA (Linear Discriminant Analysis). In this example, A represents "normal," and B to E represent the degree of edema, with severity increasing from B to E. The five-group discrimination accuracy is 63.3%. 【0091】 Furthermore, the assessment level is not limited to five stages and can be set arbitrarily as needed. Additionally, for example, a regression method may be used to estimate the viscosity coefficient as a measure of severity. Moreover, by training a machine learning model with the normal elastic modulus of the skin at the palpation site as training data, the machine learning model can differentiate between "non-pitting edema" (where no indentation remains) and "normal skin." 【0092】 (8) The control unit 1 of the edema severity presentation object characteristic estimation device PS transmits information representing the edema severity determination result to the physician terminal MT, for example, from the communication I / F unit 5. When the physician terminal MT receives the information representing the edema severity determination result, it displays the received information on the physician terminal MT's display. 【0093】 Furthermore, the physician terminal MT sets operating parameters for the tactile state presentation device MS to reproduce the patient's difficulty in reducing edema, based on the received information representing the severity of the edema. As a result, for example, when the physician presses the operation unit BT of the tactile state presentation device MS, the operation unit BT is pressed down with a hardness corresponding to the difficulty in reducing the edema, thereby allowing the physician to experience the tactile state of the patient's tissue, even if the patient is in a remote location, while remaining in the hospital. 【0094】 (Effects) As described above, in one embodiment, measurement data of finger pressure when the area to be palpated by the patient is pressed with the fingertip, and video data of the area to be palpated being pressed are acquired. Based on the RGB image data and depth image data included in the acquired video data, the region of the area to be palpated is extracted, the three-dimensional coordinates of the finger being palpated in the extracted area to be palpated are detected, and a three-dimensional normal vector is set on the area to be palpated. Then, the change in the three-dimensional coordinates of the finger being palpated on the three-dimensional normal vector is calculated as the displacement amount of the area to be palpated, and based on the calculated displacement amount and the measurement data of finger pressure, the Young's modulus and viscosity characteristics of the area to be palpated are calculated, and based on the calculated Young's modulus and viscosity characteristics, it is determined whether the area to be palpated is normal or edematous, and if it is edematous, the severity thereof. 【0095】 Therefore, it is possible to determine not only the elastic modulus but also the viscosity of the affected area, and based on these, it is possible to determine whether the affected area is normal or edematous, and if it is edematous, to determine its severity. In addition, by training a machine learning model using the elastic modulus of normal skin as training data, it is also possible to differentiate between non-pitting edema and normal skin based on the calculated elastic modulus. 【0096】 Furthermore, by indicating the appropriate target pressure in response to the acupressure motion, the applicator can apply the correct pressure to the affected area. As a result, the displacement, Young's modulus, and viscous characteristics of the affected area due to the acupressure motion can be calculated with high accuracy, enabling more reliable severity assessment. 【0097】 Furthermore, by using a pressure sensor SS that is attached to the fingernail of the palpating finger and measures finger pressure based on the deformation of the nail during acupressure, it is possible to avoid interfering with the fingertip sensation of the acupressure movement on the affected area performed by the substitute, thereby making palpation by the substitute smoother. 【0098】[Other Embodiments] (1) In one embodiment, the case of using a so-called depth camera that outputs RGB image data and depth image data was described as an example, but any device capable of acquiring data in the depth direction may be used, such as a laser radar (LiDAR) or a stereo camera, or a camera that outputs infrared image (grayscale) data may be used. 【0099】 (2) In one embodiment, a sensor of the type that is attached to the fingernail of the fingertip and detects deformation of the nail was described as an example of a pressure sensor, but other sensors may be used, for example, a sensor that obtains finger pressure by optically measuring the blood flow in the fingertip nail area, or a probe-type pressure sensor may be used. 【0100】 (3) In one embodiment, the physician terminal MT sets operating parameters for the tactile state presentation device MS to reproduce the difficulty of the patient's edema returning to normal, thereby making it operational. However, the object property estimation device PS may set the tactile state presentation device MS to operational via the network NW. 【0101】 (4) In one embodiment, the processing function of the object property estimation device PS was described using the example of a case where it is installed on a personal computer or portable terminal located in a remote location where the patient is located, but it may also be installed on an information processing device such as a server computer or personal computer located on the hospital side. In this case, a communication terminal is installed in the remote location, and the video data and finger pressure measurement data are transmitted from this communication terminal to the information processing device on the hospital side via a network NW. 【0102】 (5) Furthermore, the processing functions of the object property estimation device PS may be distributed across multiple personal computers or server computers located at remote locations, hospitals, or on the Web or cloud. 【0103】(6) In one embodiment, the case of palpation of edema was described as an example, but this invention may also be applied to palpation of skin diseases other than edema, internal organ diseases, or the degree of swelling of tissue due to bruises or internal organ diseases. Furthermore, in addition to the human body, animals, fruits, etc. may be used as measurement targets, and this invention is also applicable when measuring the state of tissue in non-living objects such as mattresses and cushions. 【0104】 (7) In one embodiment, information representing the severity determination result is transmitted from the object property estimation device PS to the physician terminal MT, and the physician terminal MT operates the tactile state presentation device MS based on the severity determination result. However, it is not necessarily required that the tactile state presentation device MS reproduce tactile sensations; it is also sufficient to transmit the Young's modulus and severity from the object property estimation device PS to the physician terminal MT, and for the physician terminal MT to display the Young's modulus and severity on a display. 【0105】 Furthermore, the configuration of each processing function of the object property estimation device PS, as well as its processing procedures and processing content, can be modified in various ways without departing from the spirit of this invention. 【0106】 Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment. 【0107】 In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. 【0108】PS...Object property estimation device CM...Depth camera SS...Pressure sensor IN...Input device DP...Display device MT...Doctor's terminal MS...Tactile state presentation device 1...Control unit 2...Program storage unit 3...Data storage unit 4...Input / output I / F unit 5...Communication I / F unit 6...Bus 11...Video data acquisition unit 12...Hand / sensor area determination unit 13...Background removal unit 14...Tactile finger three-dimensional coordinate calculation unit 15...Hand / sensor area removal unit 16...Normal vector calculation unit 17...Finger curvature radius acquisition unit 18...Pressure data acquisition unit 19...Fingertip pressure presentation unit 21...Displacement amount calculation unit 22...Young's modulus calculation unit 23...Viscous feature calculation unit 24...Severity determination unit 31...Finger curvature radius storage unit 32...Fingertip physical property value correction coefficient storage unit 33...Acupressure start point coordinate storage unit 34...Determination model storage unit

Claims

1. An object property estimation device comprising: a first processing unit that acquires first measurement data representing the finger pressure when a finger pressure motion is performed on an object, and second measurement data including information representing the three-dimensional position of the finger pressure motion on the object; a second processing unit that calculates three-dimensional position information representing the finger pressure position on the object from the second measurement data and calculates the amount of displacement of the object due to the finger pressure motion based on the calculated three-dimensional position information; a third processing unit that acquires feature quantity information representing the elasticity and viscosity of the object based on the finger pressure and the amount of displacement represented by the first measurement data; and a fourth processing unit that estimates the state of the structure of the object based on the feature quantity information representing elasticity and viscosity.

2. The object property estimation device according to claim 1, further comprising a fifth processing unit that, in response to the acupressure motion, presents to the user performing the acupressure motion information representing the acupressure pressure expressed by the first measurement data and information representing an appropriate reference acupressure pressure to be targeted in the acupressure motion.

3. An object property estimation method performed by an information processing device, comprising: a process of acquiring first measurement data representing the finger pressure when a finger pressure motion is performed on an object, and second measurement data including information representing the three-dimensional position of the finger pressure motion on the object; a process of obtaining three-dimensional position information representing the finger pressure position on the object from the second measurement data, and calculating the amount of displacement of the object due to the finger pressure motion based on the calculated three-dimensional position information; a process of acquiring feature quantity information representing the elasticity and viscosity of the object based on the finger pressure and the amount of displacement represented by the first measurement data; and a process of estimating the state of the structure of the object based on the feature quantity information representing elasticity and viscosity.

4. A program that causes a processor in the object property estimation device to execute at least one of the processes performed by each processing unit in the object property estimation device according to claim 1 or 2.

Images