Hardness measurement device, method, and program

Abstract

In one embodiment of the present invention, a hardness measurement device acquires, from force sensors respectively worn on the fingertips of two fingers of a user, force measurement values representing a finger pressure for each fingertip. The hardness measurement device acquires video data which includes information representing the three-dimensional position of each fingertip. The hardness measurement device calculates, from the video data, three-dimensional position information which represents the position of each fingertip, and, on the basis of the three-dimensional position information, calculates the distance between the fingertips. When the force measurement values are not less than zero, the hardness measurement device calculates, on the basis of a change in the distance, a pressing amount for each fingertip in an operation to grab a measurement target with the fingertips. The hardness measurement device calculates, on the basis of the force measurement values and the pressing amounts, an index value of the hardness of the measurement target. The hardness measurement device outputs the index value.

Description

Hardness measuring device, method, and program

[0001] One aspect of this invention relates to a hardness measuring device, method, and program.

[0002] In recent years, the worsening shortage of doctors and the increasing depopulation of rural areas have made it increasingly difficult to sustain medical care that relies on face-to-face consultations. As a result, various telemedicine systems utilizing information and communication technology (ICT) are being developed, and one such system is a technique for remotely palpating the hardness of a patient's tissues.

[0003] This type of technology includes, for example, a method in which a robot hand equipped with two fingers, one with an oscillator and the other with a tactile sensor, grasps a protruding part of the object to be measured (the subject's tissue) with the two fingers, and measures the hardness of the object by analyzing the vibration propagation between the two fingers (see, for example, Patent Document 1).

[0004] Hayato Takeda, et al.: "Pinch palpation for palpation AI - Stiffness identification by analysis of vibration direction / frequency," Proceedings of the 29th Annual Meeting of the Virtual Reality Society of Japan, 2C1-07, 2024.

[0005] However, the above method involves a robotic hand grasping a protruding part of the object to be measured with two fingers and analyzing the vibration propagation between the two fingers. Therefore, it has the disadvantage of not being able to measure hardness if the object does not have a protruding part. In addition, palpation using a robotic hand has high implementation costs and concerns about not being able to instill a sense of security.

[0006] This invention was made in view of the above circumstances and provides a technology that can measure the hardness of an object without using a robotic hand and regardless of whether or not the object has protrusions.

[0007] To solve the above problems, one embodiment of the hardness measuring device according to the present invention comprises a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, a fifth processing unit, and a sixth processing unit. The first processing unit acquires force measurement values ​​representing the finger pressure at each fingertip from force sensors attached to the fingertips of the user's two fingers. The second processing unit acquires video data including information representing the three-dimensional position of each fingertip. The third processing unit calculates three-dimensional position information representing the position of each fingertip from the video data and calculates the distance between each fingertip based on the three-dimensional position information. The fourth processing unit calculates the amount of indentation of each fingertip in a pinching motion of the object to be measured, based on the change in distance, when the force measurement value is zero or greater. The fifth processing unit calculates an index value of the hardness of the object to be measured based on the force measurement value and the amount of indentation. The sixth processing unit outputs the index value.

[0008] According to one aspect of this invention, when the force measurement value of each fingertip of the user is zero or greater, the amount of pressure applied by each fingertip in a pinching motion of the object to be measured is calculated based on the change in distance between each fingertip, and an index value of the hardness of the object to be measured is calculated based on the force measurement value and the amount of pressure applied. This makes it possible to measure the hardness of the object to be measured without using a robotic hand and regardless of whether the object to be measured has protrusions or not.

[0009] In other words, according to one aspect of this invention, the hardness of an object to be measured can be measured without using a robotic hand and regardless of whether or not the object to be measured has protrusions.

[0010] Figure 1 is a diagram showing an example of the overall configuration of a remote palpation system using a hardness measuring device according to one embodiment. Figure 2 is a block diagram showing an example of the hardware configuration of a hardness measuring device according to one embodiment. Figure 3 is a block diagram showing an example of the software configuration of a hardness measuring device according to one embodiment. Figure 4 is a schematic diagram illustrating an example of the finger curvature radius storage unit shown in Figure 3. Figure 5 is a schematic diagram illustrating an example of the past value storage unit for the interdigital distance between two fingers shown in Figure 3. Figure 6 is a schematic diagram illustrating an example of the initial tissue thickness storage unit shown in Figure 3. Figure 7 is a schematic diagram illustrating an example of the fingertip physical property value correction coefficient storage unit shown in Figure 3. Figure 8 is a schematic diagram showing an example of the information stored in the Young's modulus storage unit shown in Figure 3. Figure 9 is a flowchart illustrating an example of operation in one embodiment. Figure 10 is a flowchart illustrating an example of operation in one embodiment. Figure 11 is a schematic diagram illustrating an example of operation in one embodiment. Figure 12 is a schematic diagram illustrating an example of operation in one embodiment. Figure 13 is a schematic diagram illustrating an example of operation in one embodiment.

[0011] Embodiments of this invention will be described below with reference to the drawings.

[0012] [One Embodiment] (Configuration Example) (1) System Figure 1 is a diagram showing an example of the overall configuration of a remote palpation system using a hardness measuring device according to one embodiment of the present invention.

[0013] This remote palpation system involves placing a stiffness measuring device 100 at a remote location such as the patient PT's home or facility, and placing a physician terminal MT and a tactile presentation device 200 at the hospital. The stiffness measuring device 100 calculates an index value corresponding to the stiffness of the measurement target (the affected area of ​​the patient PT) based on video data from a depth camera CM and force measurements from a pressure sensor SS, and transmits the index value to the physician terminal MT at the hospital via a network NW. The physician terminal MT supports the physician DR's diagnosis by presenting the stiffness corresponding to the index value to the physician DR via the tactile presentation device 200. In other words, the remote palpation system measures the stiffness of the affected area of ​​the patient PT at a remote location, transmits the index value of the stiffness to the hospital, and presents the stiffness corresponding to the index value to the physician DR at the hospital.

[0014] Here, the pressure sensor SS is attached to the fingertip of a caregiver, family member, or nurse (also called a user) who contacts the affected area of ​​the patient (PT) on behalf of the physician (DR). At this time, the attachment position is set on the fingernail so as not to interfere with the sensation of pinching the affected area with the fingertip. The pressure sensor SS obtains a force measurement value representing finger pressure by measuring the deformation of the fingernail when the caregiver (NS) pinches the affected area with their fingertip, and outputs this force measurement value to the hardness measuring device 100, for example, via a signal cable.

[0015] The depth camera CM is a camera capable of capturing infrared images and depth images. For example, the depth camera DC may be a camera that combines multiple infrared cameras capable of capturing infrared and depth images with an RGB camera capable of capturing RGB images. Alternatively, the depth camera DC may be a camera other than a stereo camera, such as a LiDAR camera. In other words, the depth camera DC can be any camera capable of capturing both infrared and depth images. Here, the depth camera CM includes, for example, an infrared light that illuminates the target with infrared light, multiple infrared cameras that image the target from different angles, and an RGB camera that images the target. The RGB camera is used to display RGB images on the physician terminal MT and is not involved in the creation of infrared and depth images, so it may be omitted. Also, one of the multiple infrared images used to create the depth image is used as the infrared image. Such a depth camera CM outputs video data including an infrared image and a depth image obtained by imaging the target illuminated with infrared light. For example, the depth camera CM is positioned diagonally above the affected area of ​​the subject and captures images of the pinching motion of the affected area by the fingertips of the surrogate nurse (NS), outputting the video data to the hardness measuring device 100 via a signal cable. Here, each fingertip of the surrogate nurse (NS) is fitted with a pressure sensor SS, which has a retroreflective marker mk attached that reflects the irradiated light back towards the light source. Consequently, in the infrared image included in the video data, the pixels indicating the retroreflective marker mk have higher brightness than other pixels due to the reflected light from the retroreflective marker mk.

[0016] In addition to a signal cable, the pressure sensor SS and depth camera CM and the hardness measuring device 100 may be connected by a wireless interface employing a low-power wireless data transmission standard such as Bluetooth® or Wi-Fi®.

[0017] On the other hand, the physician terminal MT is configured, for example, as a personal computer and is used to display images representing the pinching motion of the affected area and index values ​​of the affected area's hardness, which are transmitted from the hardness measuring device 100 at a remote location.

[0018] Furthermore, a tactile feedback device 200 is connected to the physician terminal MT. The physician terminal MT reproduces the tactile state of the affected area by operating the tactile feedback device 200 based on the index value of the hardness of the affected area.

[0019] The tactile feedback device 200 comprises a control unit 201, a drive unit 202, and a presentation unit 203. The control unit 201 controls the drive unit 202, which drives the presentation unit 203, based on index values ​​received from the physician terminal MT. The drive unit 202 is controlled by the control unit 201 and supplies compressed air to the presentation unit 203 according to the tactile sensation of hardness. The presentation unit 203 is attached to each fingertip of the physician DR and expands in response to the compressed air supplied from the drive unit 202, thereby reproducing the tactile sensation of each fingertip of the substitute nurse (NS) on each fingertip of the physician DR.

[0020] (2) Hardness measuring device 100 Figures 2 and 3 are block diagrams showing examples of the hardware configuration and software configuration of a hardness measuring device 100 according to one embodiment of the present invention.

[0021] The hardness measuring device 100 is configured, for example, by a personal computer. Alternatively, the hardness measuring device 100 may be configured by a portable device such as a smartphone, tablet, head-mounted display (HMD), or smart glasses, or by a server computer located on the web or in the cloud.

[0022] The hardness measuring device 100 includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU). The control unit 1 is connected via a bus to a storage medium having a program storage unit 2 and a data storage unit 3, an input / output interface (hereinafter referred to as I / F) 4, and a communication I / F 5.

[0023] The input / output interface 4 is connected to a depth camera CM and a pressure sensor SS, and further to an input device IN and a display device DP. Of these, the input device IN is used for operation command instructions to the hardness measuring device 100 and for the user NS to input their own finger radius. The display device DP is used to present input data such as finger radius and output data such as hardness index values ​​to the user, who is the user NS.

[0024] The program storage unit 2 is configured, for example, as a storage medium, by combining a non-volatile memory that can be written to and read at any time, such as an SSD (Solid State Drive), and a non-volatile memory such as a ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores application programs necessary to execute various controls according to one embodiment. Hereafter, the OS and each application program will be collectively referred to as a program or computer program. To add to this, there is a memory that stores computer programs that include computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to execute the methods disclosed herein. This computer program can be implemented in commonly known forms, such as a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, and / or memory of an FPGA or ASIC.

[0025] The data storage unit 3 combines, for example, a non-volatile memory such as an SSD that allows writing and reading at any time, and a volatile memory such as RAM (Random Access Memory) as a storage medium. As shown in Figures 3 to 8, the data storage unit 3 mainly includes a finger curvature radius storage unit 31, a past value storage unit 32 for the distance between two fingers, an initial tissue thickness storage unit 33, a fingertip physical property value correction coefficient storage unit 34, and a Young's modulus storage unit 35 as storage areas.

[0026] The finger curvature radius memory unit 31 is used to store information representing the radius of each finger used by the substitute nurse (NS) to pinch the affected area, for example, the finger curvature radius of the abdominal position of each fingertip used for the pinching motion. For example, as shown in Figure 4, the finger curvature radius R of the thumb is stored in the finger curvature radius memory unit 31. ｉ And the radius of curvature R of the index finger ｉ Remember that.

[0027] The two-finger distance past value storage unit 32 stores the distance between each fingertip in chronological order. For example, as shown in Figure 5, the two-finger distance past value storage unit 32 stores the frame number of the video data and the two-finger distance in association with each other for a period of k frames in the past in chronological order. Alternatively, the two-finger distance past value storage unit 32 may use the time i [s] at the time of imaging instead of the frame number.

[0028] As shown in Figure 6, the initial tissue thickness memory unit 33 stores the initial tissue thickness (thickness of the affected area) which corresponds to the initial value of the distance between two fingers when the affected area is pinched.

[0029] The fingertip physical property value correction coefficient storage unit 34 stores correction coefficients for correcting fingertip physical property values. More specifically, the elastic modulus (Young's modulus) of the finger of the agent NS (user) in the operation of pinching the affected part with each fingertip is variable according to the contact state with the affected part. For example, under low finger pressure, the indentation is dominated by the relatively soft epidermis, whereas under high finger pressure, the epidermis is compressed and the indentation is dominated by hard tissues such as bones, so the Young's modulus of the fingertip is higher under high finger pressure than under low finger pressure. For this reason, a calibration experiment is performed in advance to calculate correction coefficients for the fingertip physical property values with respect to finger pressure and indentation amount (displacement amount), and the calculated correction coefficients are stored in the fingertip physical property value correction coefficient storage unit 34. The fingertip physical property value is the fingertip Young's modulus E ｉ and the fingertip Poisson's ratio ν ｉ , and is a value based on (1−ν ｉ ２ ) / E ｉ .

[0030] As the correction coefficients for the fingertip physical property values, as shown in an example in FIG. 7, a finger pressure coefficient A, a displacement amount coefficient a, and a coefficient b are stored in the fingertip physical property value correction coefficient storage unit 34. Each of the coefficients A, a, and b is a parameter that defines a reciprocal function that approximately represents the fingertip physical property value that changes according to the contact state with the affected part {(1−ν ｉ ２ ) / E ｉ =1 / (A exp(a·δ ｚｎ +b·F ｎ ))}. Here, δ ｚｎ is the displacement amount (indentation amount). F ｎ is the load (finger pressure).

[0031] As shown in FIG. 8, the Young's modulus storage unit 35 stores the time, the distance between two fingers, the Young's modulus of the affected part related to the thumb, and the Young's modulus of the affected part related to the index finger in association with each other. The Young's modulus of the affected part related to the thumb is the Young's modulus of the affected part calculated from the measured force value and the indentation amount of the thumb. The Young's modulus of the affected part related to the index finger is the Young's modulus of the affected part calculated from the measured force value and the indentation amount of the index finger.

[0032] Returning to Figure 3, the control unit 1, as a processing function necessary to realize one embodiment of this invention, first includes a video data acquisition unit 11, a marker detection unit 12, a palpation finger three-dimensional coordinate calculation unit 13, and a two-finger distance calculation unit 14 as an image processing system.

[0033] Furthermore, the control unit 1 includes a finger curvature radius acquisition unit 15 and a pressure data acquisition unit 16 as a finger pressure processing system.

[0034] Furthermore, the control unit 1 includes, as a calculation processing system, a push-in amount calculation unit 21, a Young's modulus calculation unit 22, and a Young's modulus aggregation unit 23.

[0035] Each of the above-mentioned processing units 11-16 and 21-23 is realized by having the hardware processor of the control unit 1 execute an application program stored in the program storage unit 2. Note that some or all of the above-mentioned processing units 11-16 and 21-23 may be realized using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit). To elaborate, the functions of each of the processing units 11-16 and 21-23 may be implemented using a circuit or processing circuitry that includes general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof, which are programmed using one or more programs stored in one or more memories, or otherwise configured to perform disclosed functions. A processor is considered a processing circuitry or circuitry because it includes transistors and other circuits. A processor may also be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0036] The video data acquisition unit 11 acquires video data that includes information representing the three-dimensional position of each fingertip of the user, or surrogate NS. For example, the video data acquisition unit 11 acquires video data from the depth camera CM via the input / output I / F 4, capturing the pinching motion of the surrogate NS's fingertips on the affected area, and temporarily stores the acquired video data in the video storage area (not shown) within the data storage unit 3. Here, the video data includes an infrared image and a depth image. Furthermore, each fingertip of the surrogate NS is fitted with a pressure sensor SS, to which a retroreflective marker mk is attached that reflects infrared light emitted from an infrared light back towards the light source. Accordingly, in the infrared image, the pixels indicating the retroreflective marker mk have higher brightness than other pixels due to the reflected light from the retroreflective marker mk. The depth image is created from two images of the target taken from different angles, and represents the distance from the depth camera CM to the target, with each pixel having a pixel value representing the distance to the target. Note that the depth image is just one example of a distance image, and may be replaced with any distance image in which each pixel represents the distance to the target. For example, instead of a depth image, a distance image obtained by measuring the distance to the target using the time of flight of pulsed light traveling back and forth between the target and the target may be used. The video data acquisition unit 11 is an example of a second processing unit.

[0037] The marker detection unit 12 detects the position of the retroreflective marker from the infrared image included in the acquired video data. For example, the marker detection unit 12 detects the position of the retroreflective marker mk in the infrared image by binarizing the infrared image, in which the pixels indicating the retroreflective marker mk have a higher brightness than other pixels, using threshold processing.

[0038] The palpation finger three-dimensional coordinate calculation unit 13 calculates the three-dimensional position coordinates of the point corresponding to the pixel indicating the position of the retroreflective marker mk on the depth image included in the acquired video data.

[0039] The two-finger distance calculation unit 14 calculates the distance between the retroreflective markers mk of each fingertip as the two-finger distance based on the calculated three-dimensional position coordinates (three-dimensional position information). Note that the marker detection unit 12, the palpation finger three-dimensional coordinate calculation unit 13, and the two-finger distance calculation unit 14 are examples of a third processing unit.

[0040] Prior to measurement, the finger curvature radius acquisition unit 15 acquires the curvature radius of the suppository position of the thumb and index finger used for pinching the affected area, which are input by the substitute NS from the input device IN, via the input / output I / F 4, and stores the acquired curvature radii of the thumb and index finger in the finger curvature radius storage unit 31.

[0041] The pressure data acquisition unit 16 acquires force measurements representing finger pressure at each fingertip from pressure sensors SS attached to the fingertips of each of the two fingers of the substitute nurse (NS). Specifically, for example, when the substitute nurse (NS) performs a pinching motion on the affected area of ​​the patient (PT), the pressure data acquisition unit 16 acquires the force measurements of finger pressure output from the pressure sensors SS via the input / output interface 4 and temporarily stores them in the storage area of ​​the data storage unit 3. The pressure data acquisition unit 16 is an example of the first processing unit.

[0042] The indentation amount calculation unit 21 calculates the indentation amount of each fingertip in a pinching motion of the affected area based on the change in the distance between the two fingers, when the force measurement value is zero or greater. For example, the indentation amount calculation unit 21 may start calculating the indentation amount when the rate at which the distance between the two fingers shortens exceeds a threshold. Specifically, for example, the indentation amount calculation unit 21 detects a situation where the force measurement value is zero or greater based on the force measurement value acquired by the pressure data acquisition unit 18. The indentation amount calculation unit 21 also detects a situation where the change in the shortening of the distance between the two fingers exceeds a threshold, based on past distances between the two fingers stored in the

[0043] The Young's modulus calculation unit 22 calculates the Young's modulus of the affected area based on the force measurement values ​​acquired by the pressure data acquisition unit 18 and the amount of indentation calculated by the indentation amount calculation unit 21 for each of the thumb and index finger during the pinching motion of the affected area.

[0044] The Young's modulus aggregation unit 23 calculates the average value of Young's modulus by aggregating the calculated Young's moduli and outputs the average value of Young's modulus as the measurement result. The average value of Young's modulus is an example of an index value of hardness. The Young's modulus calculation unit 22 and the Young's modulus aggregation unit 23 are examples of a fifth processing unit. The Young's modulus aggregation unit 23 is an example of a sixth processing unit.

[0045] (Example of operation) Next, an example of operation of the hardness measuring device 100 configured as described above will be explained using the flowcharts in Figures 9 and 10 and the schematic diagrams in Figures 11 to 13.

[0046] (Step ST1) As shown in Figure 9, prior to the pinching motion of the affected area to be measured, the substitute NS inputs the radius of curvature of his thumb and index finger, which he will use for the pinching motion, to the hardness measuring device 100 via the input device IN.

[0047] The control unit 1 of the hardness measuring device 100 acquires the input curvature radius of the palpated finger via the input / output I / F 4 under the control of the finger curvature radius acquisition unit 15, and stores the acquired curvature radius of the palpated finger in the finger curvature radius storage unit 31. In Figure 4, the curvature radius R of the thumb. ｉ 20 mm is stored as the radius of curvature R of the index finger. ｉ 15mm is recorded as the default setting.

[0048] The finger curvature radius may be estimated and stored, for example, by applying image recognition technology to the image of the finger captured by a depth camera (CM).

[0049] Subsequently, the substitute NS attaches pressure sensors SS with retroreflective markers mk to the fingernails of the thumb and index finger, as shown in Figure 11. By attaching the pressure sensors SS to the fingernails, the substitute NS can directly touch the affected area of ​​the patient PT during palpation, without hindering the sensation in the fingertips or significantly restricting the pinching motion of the affected area. Subsequently, infrared light from the infrared light of the depth camera CM is irradiated onto the affected area of ​​the patient PT. At this time, the irradiated light is reflected from the retroreflective markers mk on each fingertip towards the depth camera CM. This enables the hardness measuring device 100 to operate.

[0050] In this state, when the substitute nurse (NS) begins to pinch the affected area with each fingertip, the finger pressure at each fingertip is measured by the pressure sensor SS, and the measured force is output to the hardness measuring device 100. The control unit 1 of the hardness measuring device 100 acquires this measured force via the input / output interface 4 under the control of the pressure data acquisition unit 16.

[0051] (Step ST2) Meanwhile, the depth camera CM captures images of the pinching motion of the affected area by the fingertips of the substitute nurse (NS) while infrared light is being irradiated, and outputs the obtained video data to the hardness measuring device 100. Here, the video data includes infrared images and depth images, including reflected light from the retroreflective markers mk on each fingertip.

[0052] The control unit 1 of the hardness measuring device 100, under the control of the video data acquisition unit 11, acquires video data including infrared images and depth images from the depth camera CM via the input / output interface 4 in step ST2, and temporarily stores the acquired video data in the video storage area of ​​the data storage unit 3.

[0053] (Step ST3) The control unit 1 detects the position of the retroreflective marker mk from the infrared image under the control of the marker detection unit 12. For example, as shown in Figure 12, the marker detection unit 12 generates a binarized image g_bi by thresholding an infrared image g_IR in which the pixel regions indicating the retroreflective marker mk of each fingertip of the substitute NS have a higher brightness than other pixel regions. As a result, the marker detection unit 12 detects the central position of the pixel region with high brightness in the binarized image g_bi as the position of the retroreflective marker mk. In Figure 12, the positions of two retroreflective markers mk corresponding to the fingertips of the thumb and index finger are detected, respectively.

[0054] (Step ST4) Returning to Figure 9, the control unit 1, under the control of the palpation finger three-dimensional coordinate calculation unit 13, calculates two three-dimensional position coordinates for each point corresponding to each pixel indicating the position of the two retroreflective markers mk on the depth image included in the acquired video data.

[0055] (Step ST5) Under the control of the two-finger distance calculation unit 14, the control unit 1 calculates the three-dimensional position coordinates P of the fingertip of the thumb based on the two calculated three-dimensional position coordinates, as shown in the following equation. Ａ And the three-dimensional position coordinate P of the fingertip of the index finger. Ｂ The distance between two fingers is the distance l ｆｉｎｇｅｒ It is calculated as follows.

[0056]

[0057] Subsequently, the control unit 1, under the control of the two-finger distance calculation unit 14, calculates the two-finger distance l ｆｉｎｇｅｒ The past value of the distance between two fingers is stored in the two-finger distance storage unit 32, associated with the frame number of the infrared image.

[0058] (Step ST6) The control unit 1, under the control of the indentation amount calculation unit 21, determines the initial tissue thickness t ０ The system determines whether the initial tissue thickness is already stored in the initial tissue thickness memory unit 33. If it is already stored, steps ST7 to ST9 are skipped and the system proceeds to step ST10. If not, the system proceeds to step ST7.

[0059] (Step ST7) The control unit 1, under the control of the indentation amount calculation unit 21, determines whether the force measurement values ​​of the thumb and index finger are zero or greater. If not, it returns to step ST2; if zero or greater, it proceeds to step ST8.

[0060] (Step ST8) The control unit 1, under the control of the indentation amount calculation unit 21, calculates the distance between two fingers l ｆｉｎｇｅｒ The system determines whether the rate of change exceeds a threshold. If not, it returns to step ST2; if it exceeds the threshold, it proceeds to step ST9. Note that the distance between two fingers l ｆｉｎｇｅｒ The rate of change is the distance between two fingers l ｆｉｎｇｅｒ This is the rate at which the distance between two fingers decreases, and the rate at which the distance between two fingers decreases per unit time. ｆｉｎｇｅｒ This represents the distance that decreases. Also, the distance between two fingers l ｆｉｎｇｅｒ The rate of change may be smoothed using a moving average or similar method.

[0061] (Step ST9) The control unit 1, under the control of the indentation amount calculation unit 21, calculates the current distance between two fingers l ｆｉｎｇｅｒ Initial tissue thickness t ０This is stored in the initial tissue thickness memory unit 33. That is, according to steps ST7 to ST9, the control unit 1 satisfies the condition that the force measurement value > 0 while the distance between the two fingers l is satisfied for the action of pinching the affected area with each fingertip. ｆｉｎｇｅｒ If the rate at which it shrinks exceeds a certain threshold, then the l at that moment ｆｉｎｇｅｒ Initial tissue thickness t ０ Save it as such. Note that the initial tissue thickness t ０ The value may be saved after a certain delay time or delay distance. For example, the distance l between two fingers at the point when the threshold is exceeded and compressed by 1 mm. ｆｉｎｇｅｒ Initial tissue thickness t ０ That is also acceptable.

[0062] Here, the relationship between steps ST7 to ST9 and the pinching motion of the affected area will be explained using Figure 13. There are two types of pinching motions: OP1, which involves pinching the entire protruding affected area (convex part) of the patient's physical therapy (PT), and OP2, which involves pinching a part of the flat affected area of ​​the patient's PT. The former shows the pinching motion when the object to be measured has a convex part. The latter shows the pinching motion when the object to be measured does not have a convex part. Steps ST7 to ST9 can be used in either case, regardless of whether the object to be measured has a convex part or not.

[0063] In the case of gripping the protrusion in OP1, when the fingertips fg of the substitute NS grip the protrusion, the force measurement value of the pressure sensor SS becomes zero or greater, and the distance between the two fingers l ｆｉｎｇｅｒ The rate at which it shrinks exceeds the threshold. Therefore, in the case of sandwiching the protrusion, OP1 proceeds in the order of steps ST7: Yes, ST8: Yes, ST9. That is, in the case of sandwiching the protrusion, OP1, the force measurement value is zero or greater when the distance between the two fingers l ｆｉｎｇｅｒ As it contracts, the start of the pinching motion is detected.

[0064] On the other hand, in the case of partially pinching, OP2, since there is no protrusion on the affected area, the following actions are performed sequentially: OP21, where each fingertip fg of the substitute nurse pushes both sides of the affected area; and OP22, where each fingertip fg pinches the affected area that has been temporarily made protruding by the pushing action OP21.

[0065] During the action OP21 of pressing on the affected area, the force measurement value of the pressure sensor SS becomes zero or greater, while the distance between the two fingers l ｆｉｎｇｅｒThis remains constant. Therefore, in the action OP21 of pressing on the affected area, the process proceeds in the order of step ST7: Yes, ST8: No. In other words, in the action OP21 of pressing on the affected area, although the force measurement value is greater than or equal to zero, the distance between the two fingers l ｆｉｎｇｅｒ Since it doesn't shrink, the start of the pinching motion is not detected.

[0066] Next, during the pinching motion OP22 of the affected area, the force measurement value of the pressure sensor SS is maintained at zero or greater, and the distance between the two fingers l ｆｉｎｇｅｒ The rate at which the distance between the fingers decreases exceeds the threshold. Therefore, in the pinching motion OP22 of the affected area, the process proceeds in the order of steps ST7: Yes, ST8: Yes, ST9. That is, in the pinching motion OP22 of the affected area, the distance between the two fingers is zero or greater when the force measurement value is zero or greater. ｆｉｎｇｅｒ As the gripping area contracts, the start of the pinching motion is detected. Note that, when partially gripping, distinguishing between the pressing motion OP21 and the pinching motion OP22 is preferable from the viewpoint of accurately detecting the start of the pinching motion, but it is not essential. For example, in the case of a typical operative NS, the transition from the pressing motion OP21 to the pinching motion OP22 is very short, so even if the start of the pressing motion OP21 is mistakenly detected as the start of the pinching motion, it has little effect on the Young's modulus measurement results described later.

[0067] Thus, in both cases, OP1 (when there is a protrusion) and OP2 (when there is no protrusion), when the pinching motion of the affected area is initiated, the moment l ｆｉｎｇｅｒ Initial tissue thickness t ０ Save as (step ST9). After step ST9 is completed, steps ST2 to ST6 are executed in order, and then the process moves from step ST6 to step ST10.

[0068] (Step ST10) The control unit 1, under the control of the indentation amount calculation unit 21, determines whether the force measured values ​​of the thumb and index finger are zero or greater. If not, the pinching operation has ended, and the system proceeds to step ST16. On the other hand, if the force measured values ​​are zero or greater, the system proceeds to step ST11 because the pinching operation is still in progress.

[0069] (Step ST11) The control unit 1, under the control of the indentation amount calculation unit 21, determines the initial tissue thickness t ０ The initial tissue thickness is read from the initial tissue thickness memory unit 33.

[0070] (Step ST12) The control unit 1, under the control of the indentation amount calculation unit 21, determines the initial tissue thickness t ０ And the current distance between two fingers l ｆｉｎｇｅｒ Based on this, the amount of pressure δ in each finger Ａ , δ Ｂ Calculate the distance l between two fingers at time i. ｆｉｎｇｅｒ to l ｆｉｎｇｅｒ (i) Let this be the case. At this time, the amount of pressure each finger has at time i δ Ａ , δ Ｂ And the initial tissue thickness t ０ And the distance between two fingers l ｆｉｎｇｅｒ Between (i) and δ Ａ +δ Ｂ = t ０ -l ｆｉｎｇｅｒ The relationship (i) holds. Therefore, the amount of pressure applied by the two fingers {t} ０ -l ｆｉｎｇｅｒ (i) is divided in any ratio, and the amount of pressure each finger presses down δ Ａ , δ Ｂ To estimate this, for example, as shown in equation (1) below, the amount of indentation δ of each finger Ａ , δ Ｂ You can also divide it in half.

[0071]

[0072] Alternatively, the displacement amount δ shown in equation (2) ｚｎ Based on the calculation formula, the radius of curvature R ｉ The amount of indentation δ depends on the ratio of the reciprocals of the 1 / 3 power of δ. Ａ , δ Ｂ Alternatively, the amount of indentation δ can be calculated based on a coefficient determined by experiment. Ａ , δ Ｂ You may divide it proportionally.

[0073]

[0074] However, v ｉ : Poisson ratio of the finger, v ｏ : Poisson's ratio of the object (affected area), E ｉ : Young's modulus of the finger, E ｏ : Young's modulus of the object (affected area), F ｎ : Load (force measurement value), Rｉ : refers to the radius of curvature.

[0075] In any case, the control unit 1 sends the calculated indentation amount δ under the control of the indentation amount calculation unit 21 Ａ , δ Ｂ to the Young's modulus calculation unit 22.

[0076] (Step ST13) Under the control of the Young's modulus calculation unit 22, the control unit 1 reads the finger pressure coefficient A, the displacement amount coefficient a, and the coefficient b from the fingertip physical property value correction coefficient storage unit 34. Each of the coefficients A, a, and b is a parameter that defines a reciprocal function that approximately represents the fingertip physical property value on the left side that changes according to the contact state with the affected part, as shown in the following equation (3).

[0077]

[0078] (Step ST14) Under the control of the Young's modulus calculation unit 22, the control unit 1 uses the measured force value F of each finger ｎ and the indentation amount δ of each finger Ａ , δ Ｂ and the radius of curvature R of each finger, to calculate the Young's modulus E of the affected part for each finger, as shown in equation (4) based on Hertz's contact theory ｉ . That is, the control unit 1 calculates the Young's modulus E of the affected part for the thumb ｏ and the Young's modulus E of the affected part for the index finger ｏ . Note that although equation (4) assumes contact between a sphere (finger) and an infinite plane (affected part), contact between a sphere (finger) and a sphere or an arbitrary curved surface (affected part) may also be assumed. ｏ

[0079]

[0080] In equation (4), for the thumb, the displacement amount δ ｚｎ is set as the thumb indentation amount δ Ａ , the load F ｎ is set as the measured force value F of the thumb Ａ , and the finger radius of curvature R ｉ is set as the finger radius of curvature R of the thumb Ａ . On the other hand, for the index finger, the displacement amount δ ｚｎ is set as the index finger indentation amount δ Ｂ , the load F ｎ is set as the measured force value F of the index finger Ｂ ​and the finger curvature radius R ｉ is defined as the finger curvature radius R of the index finger Ｂ . The fingertip physical property value {(1-v ｉ ２ ) / E ｉ} is approximated by a reciprocal function of formula (3). Note that the fingertip physical property value {(1-v ｉ ２ ) / E ｉ} is not limited to the reciprocal function shown in formula (3), and may be approximated by other functions. Supplementally, since the fingertip physical property value varies depending on the indentation state, a calibration experiment targeting a test piece with a known physical property value yields the displacement amount δ ｚｎ and the load F ｎ are approximated by the reciprocal of the function f(δ ｚｎ , F ｎ ). As the function f(δ ｚｎ , F ｎ ), for example, an exponential function (A exp(a·δ ｚｎ +b·F ｎ )) can be cited.

[0081] (Step ST15) Under the control of the Young's modulus calculation unit 22, as shown in FIG. 8, the control unit 1 stores the Young's modulus E of the affected part related to the thumb ｏ and the Young's modulus E of the affected part related to the index finger ｏ , the time i, and the two-finger distance l ｆｉｎｇｅｒ (i) in association with each other in the Young's modulus storage unit 35. After step ST15 ends, the process proceeds to step ST2. Thereafter, similarly, the control unit 1 calculates two Young's moduli E of the affected part for each of the thumb and the index finger through steps ST2 to ST6 and ST10 to ST15 ｏ , and stores each Young's modulus E ｏ , the time i, and the two-finger distance l ｆｉｎｇｅｒ (i) in association with each other in the Young's modulus storage unit 35. In this way, the Young's modulus of the affected part corresponding to each fingertip of the pinching operation can be obtained in time series.

[0082] (Step ST16) The control unit 1, under the control of the Young's modulus calculation unit 23, calculates the Young's modulus by reading the Young's modulus of the affected area corresponding to each finger from the Young's modulus storage unit 35. Thereafter, under the control of the Young's modulus calculation unit 23, the control unit 1 calculates the average value of the Young's modulus of the affected area corresponding to each finger by averaging the results of the Young's modulus calculation. Specifically, for example, the control unit 1 calculates the Young's modulus in the interval where the change in Young's modulus is stable, or at the end of the pinching motion (distance between two fingers l ｆｉｎｇｅｒ The average Young's modulus during the period in which it is minimized is obtained as the measurement result. This is because, depending on various factors such as the way the pinching motion is performed and whether the affected area is flat, the amount of tissue involved at the start of the pinching motion may be small, making it impossible to obtain a reliable Young's modulus of the affected area.

[0083] (Step ST17) The control unit 1, under the control of the Young's modulus calculation unit 23, outputs the average value of the Young's modulus of the affected area for each finger and transmits the average value of the Young's modulus of the affected area for each finger to the physician terminal MT via the communication I / F 5. Based on the received average value of the Young's modulus of the affected area, the physician terminal MT sets operating parameters for the tactile presentation device 200 to reproduce the tactile sensation of each finger of the substitute nurse. The presentation unit 203 of the tactile presentation device 200 is attached to the thumb and index finger of the physician DR. As a result, for example, the tactile sensation of the hardness of the affected area is presented to the physician DR's thumb and index finger from the presentation unit 203 of the tactile presentation device 200. This makes it possible for the physician DR to experience the tactile state of the affected area of ​​a patient in a remote location while inside the hospital.

[0084] (Effects) As described above, according to one embodiment, the control unit 1 acquires force measurement values ​​representing finger pressure at each fingertip from pressure sensors SS attached to the fingertips of each of the user's two fingers. The control unit 1 acquires video data including information representing the three-dimensional position of each fingertip. The control unit 1 calculates three-dimensional position information representing the position of each fingertip from the video data, and based on the three-dimensional position information, calculates the two-finger distance l between each fingertip. ｆｉｎｇｅｒ The control unit 1 calculates the distance l between two fingers if the force measurement value is zero or greater. ｆｉｎｇｅｒ Based on the changes, the amount of pressure each fingertip presses in when pinching the object to be measured is δ. Ａ , δ ＢThe control unit 1 calculates the force measurement value and the amount of indentation δ. Ａ , δ Ｂ Based on this, an index value for the hardness of the object to be measured is calculated. The control unit 1 outputs the index value.

[0085] Therefore, when the force measurement value of each fingertip of the user is zero or greater, the distance between two fingers l ｆｉｎｇｅｒ Based on the changes, the amount of pressure δ when pinching the object to be measured with each fingertip is measured. Ａ , δ Ｂ The force measurement value and the amount of indentation δ are calculated. Ａ , δ Ｂ Based on this, an index value for the hardness of the object to be measured is calculated. This makes it possible to measure the hardness of the object without using a robotic hand and regardless of the presence or absence of protrusions on the object. Therefore, the hardness parameter of the affected tissue can be easily obtained by pinching the affected area with the fingers during palpation. Furthermore, since hardness measurement is possible even if there are no protrusions on the affected area and it is not necessary to grasp the entire protrusion of the affected area, it is possible to treat a wide range of diseases, such as embedded lesions.

[0086] Furthermore, according to one embodiment, the control unit 1 controls the distance between two fingers l ｆｉｎｇｅｒ When the rate at which it shrinks exceeds a threshold, the amount of indentation δ Ａ , δ Ｂ The calculation can be started. In this case, in addition to the effects described above, the start of the pinching motion can be accurately detected, so for example, when pinching a part of the affected area, the amount of indentation δ can be detected at the start of the pressing motion OP21. Ａ , δ Ｂ This avoids unnecessary calculations such as calculating [the specified value], thereby reducing the computational load on the computer.

[0087] Furthermore, by using a pressure sensor SS that is attached to the fingernail of the palpating finger and measures finger pressure based on the deformation of the nail during the pinching motion, it is possible to avoid interfering with the fingertip sensation of the assisted nurse (NS) when pinching the affected area, thereby enabling the assisted nurse to perform palpation of the affected area more smoothly. In addition, since the assisted nurse can perceive the hardness of the affected area and the hardness measurement device 100 can digitize the information on the hardness of the affected area, it is possible to share the sensation of hardness via the tactile presentation device 200. Moreover, since a pressure sensor SS attached to the nail is used, hardness can be obtained by measurement by human hand rather than by a robotic hand.

[0088] However, since the pressure sensor SS in one embodiment is attached to the fingernail, it can only measure the vertical force (finger pressure) applied to the fingertip. That is, when gripping a part of the affected area OP2, it cannot distinguish between the finger pressure generated between the action of pressing both sides of the affected area OP21 and the action of gripping the protrusion OP22. For this reason, in one embodiment, the distance between the two fingers l ｆｉｎｇｅｒ The distance between two fingers is constantly measured. ｆｉｎｇｅｒ The point at which the distance between the two fingers begins to decrease is considered the start of the pinching motion OP22. ｆｉｎｇｅｒ The initial thickness of the tissue t ｏ And the distance between the two fingers thereafter is l ｆｉｎｇｅｒ The amount of pressure δ for each finger is calculated by dividing the shortened portion by the amount of pressure. Ａ , δ Ｂ The force is calculated, and the Young's modulus of the affected area is calculated using Hertz's contact theory. Immediately after the start of the pinching motion OP22, the affected area is not sufficiently wrapped between the two fingers, which can lead to unstable force measurements and inaccurate Young's modulus calculations. Therefore, after the completion of a series of trials, it is preferable to average the Young's modulus in the stable interval or near the end of the pinching motion, and output the resulting average Young's modulus as the measurement result.

[0089] [Other Embodiments] (1) In one embodiment, the thumb and index finger were used to pinch the object to be measured, but this is not limited to this. Any two fingers can be used as long as the pads of the fingertips face each other, such as the thumb and middle finger, thumb and ring finger, or thumb and little finger.

[0090] (2) In one embodiment, the finger position was detected from an infrared image using a retroreflective marker mk or an infrared light, but the method is not limited to this, and the finger position may be detected by any method such as an object recognition AI, a skeletal estimation model, or an AR (Augmented Reality) marker.

[0091] (3) In one embodiment, multiple infrared cameras were used to acquire infrared images and depth images, but the invention is not limited to this, and one infrared camera may be used to acquire infrared images and multiple RGB cameras may be used to acquire depth images. Also, in one embodiment, the case in which a depth camera CM that outputs infrared images and depth images was used was explained as an example, but as a type of depth camera, laser lidar (LiDAR) or stereo camera may be used, or a camera that outputs RGB image data may be used. Accordingly, if the infrared camera is omitted, the finger position may be detected by a method other than infrared imaging, as in (2) above.

[0092] (4) In one embodiment, the example given was that the depth camera CM is equipped with an infrared light, but the invention is not limited to this, and the infrared light may be omitted from the depth camera CM by placing a separate infrared light near the depth camera CM.

[0093] (5) In one embodiment, a sensor of the type that is attached to the fingernail of the fingertip and detects deformation of the nail was described as an example of a pressure sensor, but other than that, for example, a sensor that obtains finger pressure by optically measuring the blood flow in the fingertip nail area may be used, or a probe type pressure sensor may be used.

[0094] (6) In one embodiment, when calculating the Young's modulus of the affected area based on Hertz's contact theory, the radius of curvature R of the finger ｉ In this example, we assumed contact between a sphere (finger) and an infinite plane (affected area), but we are not limited to this; we may assume contact between a sphere and an infinite plane other than a sphere. For example, when calculating the Young's modulus of the affected area based on Hertz's contact theory, we may use the respective radii of curvature of the finger and the affected area to assume contact between two spheres (finger and affected area), or contact between arbitrary curved surfaces (finger and affected area).

[0095] (7) In one embodiment, the reciprocal function of the reciprocal of the exponential function was used as an approximation method for the fingertip physical properties, but the method is not limited to this, and the reciprocal function of the reciprocal of a function other than the exponential function may be used, or an approximation method other than the reciprocal function may be used.

[0096] (8) In one embodiment, the fingertip physical properties were approximated by the reciprocal function, but the invention is not limited to this, and personalized fingertip physical properties may be used for a specific proxy NS, or fingertip physical properties obtained from the results of calibration experiments on multiple people may be used for proxy NS other than those multiple people.

[0097] (9) In one embodiment, the affected area of ​​the human body, which is a patient's physical therapist, was used as the measurement target. However, the invention is not limited to this, and parts of the human body for cosmetic purposes, such as the skin of the face or the soles of the feet, may also be used as the measurement target. Furthermore, the hardness of animals or plants other than the human body may be used as the measurement target, and this invention is also applicable when measuring the hardness of non-living objects such as cushions or food.

[0098] (10) In one embodiment, the physician terminal MT sets operating parameters for the tactile presentation device 200 to reproduce the hardness of the affected area and makes it operational. However, the hardness measuring device 100 may set the tactile presentation device 200 to an operational state via the network NW. That is, the hardness measuring device 100 may be configured to control the tactile presentation device 200 without going through the physician terminal MT.

[0099] (11) In one embodiment, a tactile presentation device 200 of the type in which a presentation unit 203 is attached to the fingertip of a physician (DR) was used, but the invention is not limited to this, and a tactile presentation device of the type that simulates the affected area of ​​a patient may be used, and when the physician's fingertip makes a pinching motion of the affected area simulated by the tactile presentation device, the tactile presentation device may be configured to provide tactile feedback to the physician's fingertip.

[0100] (12) In one embodiment, the Young's modulus is transmitted from the hardness measuring device 100 to the physician terminal MT, and the physician terminal MT operates the tactile presentation device 200 based on the Young's modulus. However, it is not necessarily required that the tactile presentation device 200 reproduce tactile sensations; the physician terminal MT that receives the Young's modulus may simply display the Young's modulus on a display.

[0101] (13) In one embodiment, Young's modulus was used as an indicator of hardness, but the present invention is not limited to this, and the present invention is also included in the scope of using Young's modulus as an indicator of softness.

[0102] (14) In one embodiment, the processing function of the hardness measuring device 100 was described using the example of a case where it is installed on a personal computer or portable terminal located in a remote location where the patient PT is located, but it may also be installed on an information processing device such as a server computer or personal computer located on the hospital side. In this case, a communication terminal is installed in the remote location, and the communication terminal is configured to transmit video data and data such as force measurement values ​​to the information processing device on the hospital side via a network NW.

[0103] (15) The processing functions of the hardness measuring device 100 may also be distributed among multiple personal computers or server computers located at remote locations, hospitals, or on the Web or cloud.

[0104] (16) In one embodiment, the hardness measuring device 100 was located at a distance from the physician terminal MT, but this is not limited to this. For example, the hardness measuring device 100 may be placed near the physician terminal MT, the physician DR may use the hardness measuring device 100 to palpate the affected area, and the Young's modulus of the affected area obtained may be displayed on the display of the physician terminal MT.

[0105] Furthermore, the configuration of each processing function of the hardness measuring device 100, as well as its processing procedures and processing contents, can be modified in various ways without departing from the spirit of this invention.

[0106] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0107] In short, this invention is not limited to the embodiments described above, and in the implementation stage, the components can be modified and materialized without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined.

[0108] CM...Depth camera SS...Pressure sensor mk...Retroreflective marker IN...Input device DP...Display device MT...Doctor's terminal NS...Proxy DR...Doctor 1,201...Control unit 2...Program storage unit 3...Data storage unit 4...Input / output I / F 5...Communication I / F 11...Video data acquisition unit 12...Marker detection unit 13...Tactile finger three-dimensional coordinate calculation unit 14...Two-finger distance calculation unit 21...Indentation amount calculation unit 22...Young's modulus calculation unit 23...Young's modulus aggregation unit 31...Finger curvature radius storage unit 32...Two-finger distance past value storage unit 33...Initial tissue thickness storage unit 34...Fingertip physical property value correction coefficient storage unit 100...Hardness measuring device 200...Tactile presentation device 202...Drive unit 203...Presentation unit

Claims

1. A hardness measuring device comprising: a first processing unit that acquires force measurement values ​​representing finger pressure at each fingertip from force sensors attached to the fingertips of two fingers of a user; a second processing unit that acquires video data including information representing the three-dimensional position of each fingertip; a third processing unit that calculates three-dimensional position information representing the position of each fingertip from the video data and calculates the distance between each fingertip based on the three-dimensional position information; a fourth processing unit that, when the force measurement value is zero or greater, calculates the amount of indentation of each fingertip in a pinching motion of the object to be measured based on the change in distance; a fifth processing unit that calculates an index value of the hardness of the object to be measured based on the force measurement value and the amount of indentation; and a sixth processing unit that outputs the index value.

2. The hardness measuring device according to claim 1, wherein the fourth processing unit starts calculating the amount of indentation when the speed at which the distance decreases exceeds a threshold.

3. A method performed by a hardness measuring device, comprising: acquiring force measurement values ​​representing finger pressure at each fingertip from force sensors attached to the fingertips of two fingers of a user; acquiring video data including information representing the three-dimensional position of each fingertip; calculating three-dimensional position information representing the position of each fingertip from the video data, and calculating the distance between each fingertip based on the three-dimensional position information; if the force measurement value is zero or greater, calculating the amount of indentation of each fingertip in a pinching motion of the object to be measured based on the change in distance; calculating an index value of the hardness of the object to be measured based on the force measurement value and the amount of indentation; and outputting the index value.

4. A program that causes a processor in the hardness measuring device to execute at least one of the processes performed by each processing unit in the hardness measuring device according to claim 1 or 2.

Images