Photoacoustic imaging device and photoacoustic imaging method

Abstract

In the present invention, a control unit sets a scan path of a light source probe unit so as to include a target region designated by a vision unit, and guides and controls the distal end part of a robot tube according to the scan path.

Description

Photoacoustic Imaging Device and Photoacoustic Imaging Method

[0001] The present invention relates to a photoacoustic imaging technique for irradiating a subject with light and detecting and imaging photoacoustic waves generated from a light absorber in the subject.

[0002] In recent years, as a photoacoustic imaging technique and a photoacoustic tomography technique, a photoacoustic wave imaging device has been proposed that includes an LED light source unit that irradiates a subject with pulsed light and an ultrasonic detector that detects acoustic waves as ultrasonic waves generated by an object in the subject (see Citation Document 1).

[0003] This photoacoustic wave imaging device irradiates an object such as a blood vessel with light and measures the shape of the object non-contact at the image level resolution by detecting ultrasonic waves emitted by the excited object.

[0004] Conventionally, minimally invasive diagnostic and surgical tools such as endoscopes and laparoscopes in the medical field provide surgical access to the target site while minimizing patient trauma. For example, an endoscope is inserted into a body cavity such as the stomach or intestine and is used for observing the tissue surface in the body cavity, diagnosing by collecting diseased pieces with forceps, and performing treatment.

[0005] In recent years, as an endoscope and a laparoscope in the medical field, a robotic system by remote operation has been proposed to enable surgical access through a natural opening or a minimally invasive route (see Citation Document 2).

[0006] This robotic system has a tool that can direct at least one degree of freedom to the distal end through an elongated main body extending between the proximal end and the distal end using a control device that mechanically transmits user input to the distal operation segment, and is a direct drive endoscope method system in which the user can simultaneously control multiple degrees of freedom in combination with a frame that is movably connected to this tool and directs at least one degree of freedom.

[0007] Japanese Patent Application Laid-Open No. 2015-29550, Patent No. 6085530

[0008] Incidentally, it is conceivable to combine a remotely controlled robotic system, as described in Reference 2, with photoacoustic imaging technology. Specifically, a light source integrated probe of a photoacoustic wave imaging device is mounted on the end effector, which is the distal end of the robotic system, and the light source integrated probe is scanned over the target area of ​​the subject in response to remote control by the examiner.

[0009] However, actually mounting a light source-integrated probe on the end effector of a robotic system and scanning the target area of ​​the subject while ensuring that the distance from the surface is uniform requires a high level of technical skill, even if the examiner is dexterous.

[0010] In particular, imaging deep internal organs such as the liver, stomach, and intestines, which have traditionally been considered impossible to image beyond the skin surface area (a depth of about 15 mm from the skin), has been extremely difficult to achieve with conventional robotic systems. This requires the ability to scan the abdomen percutaneously with a light source-integrated probe without damaging or pressing it against the organs, while maintaining precise control.

[0011] This invention has been made in consideration of the above points, and aims to propose a photoacoustic imaging apparatus and a photoacoustic imaging method that can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep regions of a target area with significantly higher accuracy and while ensuring the safety of the subject, for target tissues or organs within the subject.

[0012] To solve these problems, the present invention provides a photoacoustic imaging device that detects photoacoustic waves generated from a light absorber in a subject and images the light absorber based on said photoacoustic waves, comprising: a robot tube configured to be bendable and extendable in the direction of travel from its base to its tip, and capable of being driven to guide its tip in a desired direction and position in response to external operation or autonomously; a light source probe unit provided at the tip of the robot tube, which integrally includes a light irradiation unit that irradiates pulsed light of a wavelength absorbed in the subject, and a photoacoustic wave detection unit that detects photoacoustic waves generated from the light absorber; a vision unit that, based on the image content of the adjacent area at the tip of the robot tube captured by an imaging sensor attached to the light source probe unit, specifies a target area of ​​the tissue or organ to be examined in the subject, either by external input or automatically; and a control unit connected to the base of the robot tube for controlling the robot tube, the light source probe unit, and the vision unit, respectively, wherein the control unit sets the scan path of the light source probe unit to include the target area specified by the vision unit, and guides and controls the tip of the robot tube according to the scan path.

[0013] As a result, in the photoacoustic imaging device, when a target area of ​​tissue or organ to be examined within a subject is specified, the scan path of the light source probe unit is set to include the target area, and the tip of the robot tube is guided and controlled according to that scan path. This allows the device to scan the target tissue or organ within the subject with a precision that is even higher than manual scanning, and generates highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target area.

[0014] Furthermore, in the present invention, the vision unit includes a bio-element recognition unit that sequentially recognizes tissues or organs corresponding to a target region based on the image content of the target region captured by the imaging sensor, while referring to a bio-element estimation model constructed by deep learning using surface feature patterns classified for each human tissue or organ as training data.

[0015] As a result, when specifying a target area of ​​tissue or organ to be examined within a subject, the photoacoustic imaging device can significantly improve the recognition accuracy and speed of the tissue or organ corresponding to that target area.

[0016] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject, and the control unit guides and controls the tip of the robot tube so that the distance to the surface of the biological part is maintained within a certain range based on the three-dimensional shape data detected by the RGB-D sensor.

[0017] As a result, the photoacoustic imaging device can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in a nearly homogeneous tracing state when guiding and controlling the tip of the robot tube according to the scanning path of the light source probe unit, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0018] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject; the vision unit has an internal motion detection unit that detects the pulsation of tissue or peristalsis of organs corresponding to a target region sequentially recognized by the biological element recognition unit based on the time change of the three-dimensional shape data detected by the RGB-D sensor; and the control unit is configured to guide and control the tip of the robot tube while making minute fluctuations in the opposing distance to the surface of the biological part in synchronization with the pulsation of tissue or peristalsis of organs detected by the internal motion detection unit.

[0019] As a result, the photoacoustic imaging device can generate highly reproducible, real-time, three-dimensional photoacoustic wave images of peripheral blood vessels in a more uniform tracing state when guiding and controlling the tip of the robot tube according to the scanning path of the light source probe unit, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0020] Furthermore, in the present invention, the vision unit includes a photoacoustic wave image generation unit that generates a photoacoustic wave image based on photoacoustic waves detected by a photoacoustic wave detection unit in a light source probe unit, and an abnormality state detection unit that detects abnormalities in tissues or organs corresponding to target regions sequentially recognized by a biological element recognition unit based on the photoacoustic wave image generated by the photoacoustic wave image generation unit.

[0021] In this way, photoacoustic imaging devices detect abnormal conditions (such as heterogeneity or the presence or absence of specific tissues) in the target area based on photoacoustic wave images. This allows imaging to extend to areas deeper than the target surface area (approximately 15 mm) as is possible with endoscopes and laparoscopes, enabling more advanced imaging diagnostics.

[0022] Furthermore, in the present invention, the vision unit includes a disease symptom estimation unit that estimates the disease and symptoms of a tissue or organ based on the abnormal state of the tissue or organ detected by the abnormal state detection unit, while referring to a disease content estimation model constructed by deep learning using abnormal state feature patterns classified for each human tissue or organ as training data.

[0023] In this way, photoacoustic imaging devices can estimate diseases and symptoms of tissues or organs based on the abnormal state of those tissues or organs, and by analyzing that abnormal state, it becomes possible to diagnose diseases and symptoms of tissues and organs that could not be detected by imaging of the target surface area alone, such as with endoscopy or laparoscopy, with relatively high accuracy.

[0024] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The vision unit further includes a vascular latent flow determination unit that, based on the time change of the three-dimensional shape data detected by the RGB-D sensor, detects the pulsation of the subject's arterial blood vessels and determines how easily the blood perfusion of the arterial blood vessels changes from the fluctuation range of the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial blood vessels. The disease symptom estimation unit is configured to reflect the ease with which the blood perfusion of the arterial blood vessels changes, as determined by the vascular latent flow determination unit, in the analysis of the abnormal state of tissue or organ.

[0025] Thus, photoacoustic imaging devices can also detect abnormalities in tissues and organs caused by changes in blood perfusion in the subject's arterial blood vessels.

[0026] Furthermore, in the present invention, the vision unit includes an optical acoustic wave image generation unit that generates an optical acoustic wave image based on optical acoustic waves detected by an optical acoustic wave detection unit in the light source probe unit, and an optical acoustic wave image evaluation unit that evaluates the clarity of the optical acoustic wave image generated by the optical acoustic wave image generation unit for each scan path. The control unit, when the clarity of the optical acoustic wave image evaluated by the optical acoustic wave image evaluation unit is below a predetermined level, causes the light source probe unit to scan while providing feedback-like guidance control to the tip of the robot tube according to the corresponding scan path.

[0027] In this way, with the photoacoustic imaging device, if the subject experiences micro-vibrations such as tremors or spasms during the sliding movement of the light source probe unit, resulting in an unclear photoacoustic wave image of the tissue or organ corresponding to the target area, the device automatically rescans the area. This allows for the stable generation of three-dimensional photoacoustic wave images of peripheral blood vessels, even if they have a three-dimensional shape like an organ.

[0028] Furthermore, in the present invention, in a photoacoustic imaging method that detects photoacoustic waves generated from a light absorber in a subject and images the light absorber based on said photoacoustic waves, a light source probe unit is provided at the tip of a robot tube that is bendable from its base to its tip and expandable and contractible in the direction of travel, and can be driven to guide its tip in a desired direction and position in response to external operation or autonomously. The probe unit integrally includes a light irradiation unit that irradiates pulsed light of a wavelength absorbed in the subject and a photoacoustic wave detection unit that detects photoacoustic waves generated from a light absorber. The method comprises a first step of specifying, by external input or automatically, a target area of ​​the tissue or organ to be examined in the subject, based on the image content of the adjacent area at the tip of the robot tube, which is imaged by an imaging sensor attached to the light source probe unit, and a second step of setting a scan path of the light source probe unit to include the target area specified in the first step, and guiding and controlling the tip of the robot tube according to the scan path.

[0029] As a result, in the photoacoustic imaging method, when a target area of ​​tissue or organ to be examined within a subject is specified, the scan path of the light source probe unit is set to include the target area, and the tip of the robot tube is guided and controlled according to that scan path. This allows the target tissue or organ within the subject to be scanned with a precision that is even higher than manual scanning, and enables the generation of highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target area.

[0030] According to the present invention, it is possible to realize a photoacoustic imaging apparatus and a photoacoustic imaging method that can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep regions of a target area of ​​tissue or organ with significantly higher accuracy than conventional methods while ensuring safety for the subject.

[0031] This is an external perspective view showing the configuration of the photoacoustic imaging device according to this embodiment. This is a schematic diagram showing the configuration of the robot tube shown in Figure 1. This is a conceptual diagram showing the configuration of the light source probe unit shown in Figure 1. This is a conceptual diagram representing the model structure for deep learning. This is a block diagram showing the internal configuration of the control system in the photoacoustic imaging device. This is a conceptual diagram showing the functional configuration of the vision unit in another embodiment.

[0032] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0033] (1) Diagram 1 of the configuration of the photoacoustic imaging apparatus according to this embodiment is a schematic external view of the photoacoustic imaging apparatus 1 according to this embodiment. The photoacoustic imaging apparatus 1 consists of a robot tube 2, a light source probe unit 3 provided at the tip of the robot tube 2, a vision unit 4 attached to the light source probe unit 3, and a control unit 5 for controlling the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0034] The robot tube 2 is configured to be bendable and extendable in the direction of travel from its base to its tip, and is configured to be driveable to guide its tip to a desired direction and position in response to external operation or autonomously.

[0035] The light source probe unit 3 is provided at the tip of the robot tube 2 and integrally includes a light irradiation unit 10 that irradiates pulsed light of a wavelength absorbed within the subject, and a photoacoustic wave detection unit 11 that detects photoacoustic waves generated from a light absorber.

[0036] The vision unit 4, based on the imaging data of the area near the tip of the robot tube 2 captured by the imaging sensor 12 attached to the light source probe unit 3, specifies the target area of ​​the tissue or organ to be examined in the subject, either via external input or automatically.

[0037] The vision unit 4 has only the imaging sensor 12 attached to the tip of the robot unit 2, alongside the light source probe unit 3; all other functional components are housed within the same housing as the control unit 5. The imaging sensor 12 is electrically connected to the main body of the vision unit 4 via the robot unit 2 and the universal cord 15.

[0038] The vision unit 4 is connected to the base end of the robot tube 2 via a universal cord 15 and has a photoacoustic wave image generation unit 20 that processes the photoacoustic wave detected by the light source probe unit 3 to image the light absorber.

[0039] Furthermore, the vision unit 4 is connected to an image display unit 21, which consists of a liquid crystal monitor or the like, capable of simultaneously displaying the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 and the image captured by the imaging sensor 12.

[0040] The control unit 5 is housed in the same enclosure as the vision unit 4 and is connected to the base end of the robot tube 2, and is configured to control the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0041] (2) Robot Tube Configuration As shown in Figure 2(A), one end (base end) of the retractable mechanism 30 of the robot tube 2 is connected via a universal cord 15 drawn out from an input / output connector connected to the control unit 5, and the other end of the retractable mechanism 30 is connected to the bending mechanism 31.

[0042] The telescopic mechanism 30 is mounted so that the shaft 33 can slide coaxially and rotate coaxially within the tubular holder 32. Specifically, the telescopic mechanism 30 is configured such that the shaft 33 can slide linearly within the tube relative to the tubular holder 32 in response to the drive of a sliding mechanism (for example, a rack and pinion mechanism or a worm gear mechanism), and the shaft 33 can rotate coaxially with respect to the tubular holder 32 in response to the drive of a rotation mechanism (for example, a planetary gear mechanism) (Figure 2(B)).

[0043] The bending mechanism section 31 has a structure that partially applies the operation support device described in Japanese Patent No. 6734927 by the applicant of the present application. That is, the bending mechanism section 31 has an articulated structure body 35 in which links arranged in series are connected so as to be relatively rotatable, and all the links are integrally deformable so as to be bendable. While sliding the rear link in the connection direction between the links through a linear member inserted inside each link, it is driven so that the sliding direction, sliding speed, and sliding position become a desired state, and the articulated structure body 35 is caused to perform a bending operation (extension operation or bending operation) (FIG. 2(C)).

[0044] In the bending mechanism section 31, the base side that holds the articulated structure body 35 is connected to the expansion and contraction mechanism section 30, and a light source probe unit 3 is provided as an end effector at the leading link (tip portion) of the articulated structure body 35.

[0045] Thus, the robot tube 2 can guide the light source probe unit 3 as an end effector to a desired direction and position while driving it to be bendable and expandable and contractible in the traveling direction.

[0046] (3) Configuration of the light source probe unit As shown in FIG. 3, the light source probe unit 3 is provided with a light irradiation unit 10 that irradiates pulsed light having a wavelength absorbed inside the subject, and a photoacoustic wave detection unit 11 that detects photoacoustic waves generated from a photoabsorber, integrally.

[0047] In the present embodiment, the light source probe unit 3 is configured such that a pair of light irradiation units 10 sandwich the photoacoustic wave detection unit 11, and can perform inspection in either the positive direction or the reverse direction of the scan direction.

[0048] Further, in the light source probe unit 3, the irradiation port of the light irradiation unit 10 and the detection port of the photoacoustic wave detection unit 11 are attached along the longitudinal direction (link connection direction) to the tip portion in the bending mechanism section 31 of the robot tube 2.

[0049] The light irradiation unit 10 consists of multiple light-emitting diode elements (not shown) connected in series and arranged along the longitudinal direction (axial direction). The light irradiation unit 10 irradiates the subject with each pulse of light, which has an infrared wavelength (for example, a wavelength of about 850 nm) emitted from each of the multiple light-emitting diode elements in response to the current supplied from the light source drive unit 40 (Figure 5).

[0050] Furthermore, by pre-setting the wavelength of pulsed light according to the type of object being examined (light absorber such as hemoglobin, blood vessels, nerve tissue, lymph, tumors, etc.), it is possible to image only the desired object.

[0051] Then, the pulsed light irradiated onto the subject from the light irradiation section 10 of the light source probe unit 3 is absorbed by an object (light absorber) within the subject. The object expands and contracts (returns from its expanded size to its original size) in accordance with the irradiation intensity (amount absorbed) of the pulsed light, thereby generating photoacoustic waves from the object.

[0052] In this way, when the light irradiation unit 10 irradiates the target object (blood vessels, nerve tissue, tumor, etc.) of the subject in the light source probe unit 3 with light, specific molecules in the body become excited due to the absorption of light by the object, and heat is generated when they return from the excited state to the steady state. The photoacoustic wave detection unit 11 detects the photoacoustic waves (photosonic waves) of molecules generated by the temperature difference (thermal expansion) in the surrounding area.

[0053] (4) Configuration of the vision unit As shown in Figure 3 above, the vision unit 4 has an imaging sensor 12 attached to the light source probe unit 3, and while imaging the nearby area at the tip of the robot tube 2 with the imaging sensor 12, it specifies the target area of ​​the tissue or organ (liver, stomach, intestines, etc.) to be examined of the subject, either by external input or automatically, based on the image content.

[0054] The imaging sensor 12 has an RGB-D sensor 41 for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit 11 to the surface of a biological part within the subject. The RGB-D sensor 41 is a sensor that can acquire depth information from the sensor to each point in the image, in addition to texture information similar to that of a general imaging camera, and can obtain three-dimensional information of the imaging environment.

[0055] Furthermore, the vision unit 4 includes a photoacoustic wave image generation unit 20 (Figure 5) that generates a photoacoustic wave image based on the photoacoustic waves detected by the photoacoustic wave detection unit 11 in the light source probe unit 2.

[0056] This vision unit 4 includes a bio-element recognition unit 50 (Figure 5) as a means for performing functional processing on the image content based on the imaging sensor 12.

[0057] The biological element recognition unit 50 uses surface feature patterns classified by human tissue or organ as training data and, while referring to a biological element estimation model constructed by deep learning, sequentially recognizes the tissue or organ corresponding to the target region based on the image content of the target region captured by the imaging sensor.

[0058] Specifically, the biological element recognition unit 50 applies a biological element estimation model consisting of three modules—a convolutional neural network (CNN) layer, a batch normalization layer, and an activation function layer (tanh function)—and a fully connected layer, as shown in Figure 4, in order to recognize the tissue or organ of the subject.

[0059] In a biocomponent estimation model, partial surface images of tissues or organs from the current frame and the frame two frames prior can be input to calculate the likelihood for multiple types of biocomponents related to those tissues or organs.

[0060] Furthermore, the biological element recognition unit 50, as a recognition result, post-processes the data by identifying the most frequently observed biological element among the 15 frames (the current frame and the previous 14 frames) as the current tissue or organ. In this way, the biological element recognition unit 50 learned a biological element estimation model using supervised learning with its own dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0061] As a result, the photoacoustic imaging device 1 can significantly improve the recognition accuracy and speed of tissues or organs corresponding to the target area when specifying a target area of ​​tissue or organ to be examined within a subject.

[0062] (5) Internal configuration of the control system in the photoacoustic imaging device. Diagram 5 shows the internal configuration of the control system in the photoacoustic imaging device 1. The control unit is responsible for the overall control of the photoacoustic imaging device 1 and is mainly composed of a microcomputer consisting of an MCM (Multi-Chip Module) equipped with a CPU (Central Processing Unit) and memory.

[0063] The control unit 5 controls the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0064] The control unit 5 sets the scan path of the light source probe unit 3 so as to encompass the target area of ​​the subject's tissue or organ designated by the vision unit 4. In other words, the control unit 5 sets a scan path for each target area so that the target area of ​​the subject's tissue or organ to be examined, which is externally input or automatically designated by the vision unit 4, is evenly covered, and so that the light source probe unit 3 moves along the surface of the target area in a nearly non-contact manner.

[0065] This scan path is set as a path in a three-dimensional coordinate system, centered on the target area of ​​the subject's tissue or organ, based on the three-dimensional information of the imaging environment obtained by the imaging sensor 12 (RGB-D sensor 41) in the vision unit 4. The paths are mutually orthogonal to each other, with the direction of travel and the perpendicular direction of the light source probe unit 3 as mutually orthogonal axes.

[0066] Furthermore, the control unit 5 sends a trigger signal to the light source drive unit 40 in the light source probe unit 3 to drive and control the light irradiation. The light source drive unit 40 generates a DC current from power supplied from an external power source (not shown) and, based on the pulsed trigger signal from the control unit 5, switches a switch consisting of, for example, an FET (Field Effect Transistor) on or off, thereby emitting pulsed light of a predetermined width and frequency from multiple light-emitting diode elements.

[0067] Furthermore, the control unit 5 sends a sampling trigger signal to the photoacoustic wave image generation unit 20 in the vision unit 4, causing it to generate an image with a resolution corresponding to the target object (a tomographic image based on acoustic waves) in real time, and display the image on the image display unit 21.

[0068] The control unit 5 controls the telescopic mechanism 30 of the robot tube 2 to slide and rotate the bending mechanism 31, and while controlling the bending mechanism 31 to bend the light source probe unit 3 provided at its tip, it guides it to cover the target area of ​​the subject's tissue or organ according to the set scan path.

[0069] When the control unit 5 guides and controls the light source probe unit 3, it rotates the extension / retraction mechanism 30 of the robot tube 2 to orient the irradiation port of the light irradiation section 10 and the detection port of the photoacoustic wave detection section 11 of the light source probe unit 3 in the desired direction.

[0070] In other words, when the light source probe unit 3 is slid, the control unit 5 controls the orientation of the light source probe unit 3 so that the incident angle of the light irradiation section 10 in the light source probe unit 3 remains perpendicular to the sliding direction.

[0071] As a result, the photoacoustic imaging device 1 can stably generate three-dimensional photoacoustic wave images of peripheral blood vessels, even if the surface has a three-dimensional shape, by always irradiating the target area of ​​the subject's tissue or organ at a perpendicular angle.

[0072] (6) Operation and Effects of the Photoacoustic Imaging Apparatus In the above configuration, when the target area of ​​the tissue or organ to be examined of the subject is specified by external input or automatically, the control unit 5 sets the scan path of the light source probe unit 3 to include the specified target area and guides and controls the tip of the robot tube 2 according to the scan path.

[0073] As a result, the photoacoustic imaging device 1 can scan target tissues or organs within a subject with significantly higher precision than a human hand, and can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target region.

[0074] Furthermore, in the photoacoustic imaging device 1, the bio-element recognition unit 50 in the vision unit 4 uses surface feature patterns classified by human tissue or organ as training data and, while referring to a bio-element estimation model constructed by deep learning, sequentially recognizes the tissue or organ corresponding to the target region based on the image content of the target region captured by the imaging sensor 12.

[0075] As a result, the photoacoustic imaging device 1 can significantly improve the recognition accuracy and speed of tissues or organs corresponding to the target area when specifying a target area of ​​tissue or organ to be examined within a subject.

[0076] Furthermore, in the photoacoustic imaging apparatus 1, the RGB-D sensor 41 of the imaging sensor 12 in the vision unit 4 detects three-dimensional shape data, including the proximity distance from the detection port of the photoacoustic wave detection unit 11 to the surface of the biological tissue within the subject. Based on this detected three-dimensional shape data, the control unit 5 guides and controls the tip of the robot tube 2 so that the distance to the surface of the biological tissue is maintained within a certain range.

[0077] As a result, the photoacoustic imaging device 1 can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in a nearly homogeneous tracing state when guiding and controlling the tip of the robot tube 2 according to the scanning path of the light source probe unit 3, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0078] (7) Other Embodiments In the embodiments described above, the control unit 5 of the photoacoustic imaging apparatus 1 was configured to set the scan path of the light source probe unit 3 so as to include the target area of ​​the subject's tissue or organ designated by the vision unit 4. However, in addition to this, the present invention may also provide the vision unit 4 with a photoacoustic wave image evaluation unit 60 for evaluating the clarity of the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 for each scan path.

[0079] Further details will be described later, but Figure 6 shows the functional configuration of the additional functions of the vision unit 4 of the photoacoustic imaging apparatus 1 in this embodiment. The vision unit 4 has an internal motion detection unit 61 and a vascular latent flow determination unit 62 as means for performing functional processing on the image content based on the imaging sensor 12, and a photoacoustic image evaluation unit 60, an abnormality state detection unit 63 and a disease symptom estimation unit 64 as means for performing functional processing on the photoacoustic wave image based on the photoacoustic wave image generation unit 20.

[0080] Specifically, the photoacoustic wave image evaluation unit 60 analyzes the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 for each scan path to detect peripheral blood vessels within the target area of ​​the tissue or organ of the subject shown in the photoacoustic wave image.

[0081] The photoacoustic wave image evaluation unit 60 can use, for example, so-called deep learning methods, so-called template matching methods, SVM (Support Vector Machine), and machine learning methods using AdaBoost, etc., as a method for detecting peripheral blood vessels inside the target region of the tissue or organ of the subject.

[0082] Next, the photoacoustic wave image evaluation unit 60 calculates the sharpness of the edges in the images of peripheral blood vessels inside the skin and body tissue at the extremities of the detected subject, and evaluates whether the calculated sharpness is below a predetermined level.

[0083] If the clarity of the photoacoustic wave image evaluated by the photoacoustic wave image evaluation unit 60 is below a predetermined level, the control unit 5 causes the light source probe unit 3 to scan the tip of the robot tube 2 while providing feedback guidance control according to the corresponding scan path.

[0084] The predetermined level of clarity of the photoacoustic wave image at this time is set based on the examiner's visual confirmation, using a level that allows for practically sufficient visual recognition of the three-dimensional photoacoustic image of superficial peripheral blood vessels in the subject's tissue or organs.

[0085] As a result, in the photoacoustic imaging device 1, if the photoacoustic wave image of the tissue or organ corresponding to the target area becomes unclear due to the occurrence of micro-vibrations such as tremors or convulsions in the subject itself when the light source probe unit 3 slides, the device automatically rescans, thereby enabling the stable generation of a three-dimensional photoacoustic wave image of peripheral blood vessels, even if they have a three-dimensional shape such as an organ.

[0086] In the present invention, we have described a case in which the photoacoustic imaging apparatus 1 uses a control unit 5 to guide and control the tip of the robot tube 2 so that the distance between the detection port of the photoacoustic wave detection unit 11 and the surface of a biological part within the subject is maintained within a certain range, based on three-dimensional shape data detected from the vision unit 4 (RGB-D sensor 41 of the imaging sensor 12). However, in addition to this, the present invention may also provide delicate guidance control that takes into account the biological activity in the target area of ​​the subject's tissue or organ.

[0087] In other words, in the vision unit 4 of the photoacoustic imaging device 1, an internal motion detection unit 61 is provided in addition to the RGB-D sensor 41 of the imaging sensor 12. This internal motion detection unit 61 detects the pulsation of tissue or peristalsis of organs corresponding to the target region sequentially recognized by the biological element recognition unit 50, based on the time change of the three-dimensional shape data detected by the RGB-D sensor 41 of the imaging sensor 12.

[0088] Specifically, the internal motion detection unit 61 calculates the time required for one heartbeat or peristalsis from the period of the feature points of the waveform detected from the 3D shape data detected by the RGB-D sensor 41, and calculates the number of heartbeats or peristalsis by removing 60 at that time. Compared to a method that calculates the number of heartbeats or peristalsis from frequency analysis, which requires continuous measurement data for a certain period of time, the method that calculates the number of heartbeats or peristalsis from the period of the measured feature points can remove motion artifacts by removing the period of feature points that were falsely detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that of frequency analysis, and the number of heartbeats or peristalsis can be calculated from just a few heartbeats or peristalsis, making it possible to calculate the number of heartbeats or peristalsis at shorter intervals.

[0089] The control unit 5 controls the tip of the robot tube 2 by subtly varying the distance to the surface of the living body in synchronization with the pulsation of tissue or peristalsis of organs detected by the internal motion detection unit 61, thereby enabling the generation of highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in real time with even more uniform tracing.

[0090] Furthermore, in this embodiment, the photoacoustic imaging device 1 may be equipped with a function to detect abnormalities in the tissues or organs of the subject. Specifically, in the vision unit 4 of the photoacoustic imaging device 1, an abnormality detection unit 63 is provided in addition to the photoacoustic wave image generation unit 20.

[0091] The abnormality detection unit 63 detects abnormalities in tissues or organs corresponding to target regions sequentially recognized by the biological element recognition unit 50, based on the photoacoustic wave image generated by the photoacoustic wave image generation unit 20.

[0092] Specifically, the abnormality detection unit 63 analyzes and interprets the photoacoustic wave image corresponding to the target area of ​​the tissue or organ to determine the abnormality occurring in the tissue or organ (such as partial heterogeneity of the tissue or organ, the presence or absence of specific tissue, or the presence of tumors occurring in the tissue or organ), and improves the detection accuracy through the learning effect of deep learning.

[0093] In this way, the photoacoustic imaging device 1 detects abnormal conditions in tissues and organs corresponding to the target area detected based on photoacoustic wave images, enabling imaging to areas deeper than the target surface area (approximately 15 mm) as with endoscopes and laparoscopes, thus enabling more advanced imaging diagnostics.

[0094] Furthermore, in addition to the configuration of the abnormal state detection unit 63 described above, the photoacoustic imaging device 1 may also be provided with a disease symptom estimation unit 64 that reflects the detection results from the abnormal state detection unit 63. That is, in the vision unit 4 of the photoacoustic imaging device 1, the disease symptom estimation unit 64 uses abnormal state feature patterns classified for each human tissue or organ as training data, and while referring to a disease content estimation model constructed by deep learning, estimates the disease and symptoms of the tissue or organ based on the abnormal state of the tissue or organ detected by the abnormal state detection unit 63 by analyzing the abnormal state.

[0095] Specifically, the disease symptom estimation unit 64 applies a disease content estimation model consisting of three modules—a convolutional neural network (CNN) layer, a batch normalization layer, and an activation function layer (tanh function)—as shown in Figure 4 above, along with a fully connected layer, in order to estimate the disease and symptoms of the subject's tissues or organs.

[0096] The disease content estimation model takes information about the abnormal state of tissues or organs detected by the abnormal state detection unit 63 in the current frame and the frame two frames prior as input, and makes it possible to calculate the likelihood for multiple types of diseases and symptoms in tissues or organs.

[0097] Furthermore, the disease symptom estimation unit 64, as an estimation result, post-processes the data by identifying the most frequently occurring abnormal state of tissue or organ observed in 15 frames (the current frame and the previous 14 frames) as the disease or symptom related to the current tissue or organ. In this way, the disease symptom estimation unit 64 learned a disease content estimation model using supervised learning with its own dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0098] As a result, the photoacoustic imaging device 1 can diagnose diseases and symptoms of tissues and organs that could not be detected by imaging of the target surface area alone, such as with endoscopy or laparoscopy, with relatively high accuracy by estimating the disease or symptoms of the tissue or organ based on the abnormal state of the tissue or organ and analyzing the abnormal state of the tissue or organ.

[0099] Furthermore, in this embodiment, the photoacoustic imaging device 1 may be provided with a function to determine how easily the blood perfusion of the subject's arterial blood vessels can change. Specifically, in the vision unit 4 of the photoacoustic imaging device 1, the vascular latent flow determination unit 62 detects the pulsation of the subject's arterial blood vessels based on the time change of the three-dimensional shape data detected by the RGB-D sensor 41 of the imaging sensor 12, and determines how easily the blood perfusion of the arterial blood vessels can change from the range of variation in the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial blood vessels.

[0100] Specifically, the vascular latent flow determination unit 62, similar to the internal motion detection unit 61 described above, calculates the time required for one arterial pulsation from the period of the waveform feature points detected from the three-dimensional shape data detected by the RGB-D sensor 41, and calculates the number of pulsations by removing 60 at that time.

[0101] Thus, in the photoacoustic imaging device 1, the disease symptom estimation unit 64 in the vision unit 4 reflects the ease with which arterial blood flow perfusion is affected, as determined by the vascular latent flow determination unit 62, in the analysis of abnormal tissue or organ conditions, thereby enabling the analysis of abnormal tissue or organ conditions caused by the ease with which arterial blood perfusion is affected in the subject.

[0102] Furthermore, in the above-described embodiment, the photoacoustic imaging device 1 was configured to image a region (approximately 15 mm) deeper than the target surface region of the tissue or organ detected based on the photoacoustic wave image, primarily to detect the vascular condition. However, the present invention is not limited to this, and by setting the wavelength of the pulsed light emitted from the light irradiation section of the light source probe unit, it may also be possible to image tumors, metabolic processes, and other things that are not visible on the surface of the tissue or organ, which are located in a region deeper than the target surface region of the tissue or organ of the subject.

[0103] Furthermore, in the above-described embodiment, regarding the vision unit 4 of the photoacoustic imaging apparatus 1, the case described was one in which the imaging sensor 12 is positioned on the tip side (end effector side) of the light source probe unit 3. However, the present invention is not limited to this, and the light source probe unit 3 may be positioned on the tip side of the imaging sensor 12, or the light source probe unit 3 and the imaging sensor 12 may be arranged in parallel (in a positional relationship perpendicular to the longitudinal direction).

[0104] 1... Photoacoustic imaging device, 2... Robot tube, 3... Light source probe unit, 4... Vision unit, 5... Control unit, 10... Light irradiation unit, 11... Photoacoustic wave detection unit, 12... Imaging sensor, 15... Universal code, 20... Photoacoustic wave image generation unit, 21... Image display unit, 30... Telescopic mechanism unit, 31... Bending mechanism unit, 32... Tubular holder, 33... Shaft, 35... Multi-joint structure, 40... Light source drive unit, 41... RGB-D sensor, 50... Biological element recognition unit, 60... Photoacoustic wave image evaluation unit, 61... Intracellular motion detection unit, 62... Blood vessel latent flow determination unit, 63... Abnormal state detection unit, 64... Disease symptom estimation unit.

Claims

1. A photoacoustic imaging apparatus for detecting photoacoustic waves generated from a light absorber within a subject and imaging the light absorber based on said photoacoustic waves, comprising: a robot tube configured to be bendable and extendable in the direction of travel from its base to its tip, and capable of being driven to guide its tip in a desired direction and position in response to external operation or autonomously; a light source probe unit provided at the tip of the robot tube, comprising a light irradiation unit that irradiates pulsed light of a wavelength absorbed within the subject, and a photoacoustic wave detection unit that detects photoacoustic waves generated from the light absorber, all integrated together; a vision unit that, based on the image content of the adjacent area at the tip of the robot tube captured by an imaging sensor attached to the light source probe unit, specifies a target area of ​​the tissue or organ to be examined in the subject, either by external input or automatically; and a control unit connected to the base of the robot tube for controlling the robot tube, the light source probe unit, and the vision unit, respectively. The photoacoustic imaging apparatus is characterized in that the control unit sets a scan path for the light source probe unit so as to encompass the target area specified by the vision unit, and guides and controls the tip of the robot tube according to the scan path.

2. The photoacoustic imaging apparatus according to claim 1, wherein the vision unit comprises a bio-element recognition unit that sequentially recognizes tissues or organs corresponding to the target region based on the image content of the target region captured by the imaging sensor, while referring to a bio-element estimation model constructed by deep learning using surface feature patterns classified for each human tissue or organ as training data.

3. The photoacoustic imaging apparatus according to claim 1 or 2, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part in the subject, and the control unit guides and controls the tip of the robot tube so that the distance to the surface of the biological part is maintained within a certain range based on the three-dimensional shape data detected by the RGB-D sensor.

4. The photoacoustic imaging apparatus according to claim 2, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject, the vision unit includes an internal motion detection unit that detects the pulsation of tissue or peristalsis of an organ corresponding to the target region sequentially recognized by the biological element recognition unit based on the time change of the three-dimensional shape data detected by the RGB-D sensor, and the control unit guides and controls the tip of the robot tube while making minute fluctuations in the opposing distance to the surface of the biological part in synchronization with the pulsation of tissue or peristalsis of the organ detected by the internal motion detection unit.

5. The photoacoustic imaging apparatus according to claim 2, characterized in that the vision unit comprises a photoacoustic wave image generation unit that generates a photoacoustic wave image based on photoacoustic waves detected by the photoacoustic wave detection unit in the light source probe unit, and an abnormality state detection unit that detects abnormalities in tissues or organs corresponding to the target region sequentially recognized by the biological element recognition unit based on the photoacoustic wave image generated by the photoacoustic wave image generation unit.

6. The photoacoustic imaging apparatus according to claim 5, wherein the vision unit includes a disease symptom estimation unit that estimates the disease and symptoms of the tissue or organ by analyzing the abnormal state, based on the abnormal state of the tissue or organ detected by the abnormal state detection unit, while referring to a disease content estimation model constructed by deep learning using abnormal state feature patterns classified for each human tissue or organ as training data.

7. The photoacoustic imaging apparatus according to claim 6, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part in the subject, the vision unit further comprises a vascular latent flow determination unit that, based on the time change of the three-dimensional shape data detected by the RGB-D sensor, detects the pulsation of the arterial blood vessels of the subject and determines the susceptibility of blood perfusion in the arterial blood vessels from the fluctuation range of the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial blood vessels, and the disease symptom estimation unit reflects the susceptibility of blood perfusion in the arterial blood vessels determined by the vascular latent flow determination unit in the analysis of the abnormal state of the tissue or organ.

8. The photoacoustic imaging apparatus according to claim 1, wherein the vision unit comprises a photoacoustic wave image generation unit that generates a photoacoustic wave image based on photoacoustic waves detected by the photoacoustic wave detection unit in the light source probe unit, and a photoacoustic wave image evaluation unit that evaluates the clarity of the photoacoustic wave image generated by the photoacoustic wave image generation unit for each scan path, and the control unit, when the clarity of the photoacoustic wave image evaluated by the photoacoustic wave image evaluation unit is below a predetermined level, causes the light source probe unit to scan while providing feedback-like guidance control to the tip of the robot tube according to the corresponding scan path.

9. A photoacoustic imaging method for detecting photoacoustic waves generated from a light absorber in a subject and imaging the light absorber based on said photoacoustic waves, wherein a light source probe unit is provided at the tip of a robot tube that is bendable from its base to its tip and expandable and contractible in the direction of travel, and can be driven to guide its tip in a desired direction and position in response to external operation or autonomously, the probe unit integrally having a light irradiation unit that irradiates pulsed light of a wavelength absorbed in the subject and a photoacoustic wave detection unit that detects photoacoustic waves generated from the light absorber, the method comprising: a first step of specifying, by external input or automatically, a target area of ​​tissue or organ to be examined in the subject, based on the image content of a nearby area at the tip of the robot tube, which is imaged by an imaging sensor attached to the light source probe unit; and a second step of setting a scan path of the light source probe unit to include the target area specified in the first step, and controlling the guidance of the tip of the robot tube according to the scan path.

10. The photoacoustic imaging method according to claim 9, characterized in that, in the first step, the method sequentially recognizes tissues or organs corresponding to the target region based on the image content of the target region captured by the imaging sensor, while referring to a bio-element estimation model constructed by deep learning using surface feature patterns classified by human tissue or organ as training data.

11. The photoacoustic imaging method according to claim 9 or 10, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part in the subject, and in the second step, the tip of the robot tube is guided and controlled based on the three-dimensional shape data detected by the RGB-D sensor so that the distance to the surface of the biological part is maintained within a certain range.

12. The photoacoustic imaging method according to 10, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part in the subject, and comprises a third step of detecting the pulsation of tissue or peristalsis of an organ corresponding to the target region sequentially recognized in the first step based on the time change of the three-dimensional shape data detected by the RGB-D sensor, and in the second step, the tip of the robot tube is guided and controlled while making minute fluctuations in the opposing distance to the surface of the biological part in synchronization with the pulsation of tissue or peristalsis of the organ detected in the third step.

13. The photoacoustic imaging method according to claim 10, comprising: a fifth step of generating a photoacoustic wave image based on photoacoustic waves detected by the photoacoustic wave detection unit in the light source probe unit; and a sixth step of detecting abnormal conditions of tissues or organs corresponding to the target region sequentially recognized in the first step, based on the photoacoustic wave image generated in the fifth step.

14. The photoacoustic imaging method according to claim 13, further comprising a seventh step of estimating the disease or symptoms of the tissue or organ based on the abnormal state of the tissue or organ detected in the sixth step, while referring to a disease content estimation model constructed by deep learning using abnormal state feature patterns classified for each human tissue or organ as training data.

15. The photoacoustic imaging method according to 14, wherein the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part in the subject, and further comprises an eighth step of detecting the pulsation of the arterial blood vessels of the subject based on the time change of the three-dimensional shape data detected by the RGB-D sensor, and determining the susceptibility of blood perfusion of the arterial blood vessels to change from the fluctuation range of the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial blood vessels, and in the seventh step, the susceptibility of blood perfusion of the arterial blood vessels to change determined in the eighth step is reflected in the analysis of the abnormal state of the tissue or organ.

16. The photoacoustic imaging method according to claim 9, further comprising a ninth step of evaluating the clarity of a photoacoustic wave image based on a photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit for each scan path, wherein in the second step, if the clarity of the photoacoustic wave image evaluated in the ninth step is below a predetermined level, the tip of the robot tube is guided and controlled in a feedback manner according to the corresponding scan path, and the light source probe unit is made to scan.

Images