Mobile skin diagnosis device

Abstract

The present invention relates to a mobile skin diagnosis device which enables real-time observation and diagnosis of not only the epidermis but also the dermal layer and facilitates prescription in dermatology or aesthetic clinics without difficulty, thereby reducing the deviation of medical treatment results due to different levels of experience, enabling physicians to perform precise medical treatment, and significantly reducing the risk of medical accidents for patients.

Description

mobile skin diagnostic device

[0001] The present invention relates to a mobile skin diagnostic device, and more particularly to a mobile skin diagnostic device capable of three-dimensional precise measurement down to the dermis and subcutaneous tissue using a camera.

[0002] The matters described in this background technology section are written to aid in understanding the background of the invention and may include matters that are not prior art already known to those skilled in the art to which this technology belongs.

[0003]

[0004] In general, a dermatologist's clinical experience is crucial for the treatment of skin diseases or cosmetic dermatological procedures. In particular, if a patient has received Botox or fillers injected beneath the epidermis during past treatments, it is not easy to detect this superficially.

[0005] An experienced physician can relatively accurately assess a patient's skin condition through clinical intuition and judgment accumulated from numerous cases.

[0006] On the other hand, residents or doctors with limited experience find it difficult to rely on intuitive judgment due to a lack of sufficient experience, which can lead to variations in diagnosis and procedure outcomes.

[0007] Therefore, there is a need for a technical solution that allows for real-time observation of not only the epidermis but also the dermis layer in dermatological or aesthetic procedures. This will minimize variations in treatment outcomes based on the doctor's level of experience and enable the achievement of more accurate and consistent treatment results.

[0008]

[0009] Meanwhile, existing skin diagnostic devices have the following problems.

[0010] In the case of a device that photographs the entire face, it is possible to observe the condition of the epidermis and the area immediately below the epidermis within 1 mm using a UV light source and a white LED, but it is impossible to measure the dermis layer, which is the reference point for procedures such as Botox or fillers.

[0011] Furthermore, existing devices capable of performing precise skin diagnosis on the entire face are very large, which can be problematic for general users, and their price is also prohibitive for users to bear.

[0012] In the case of localized imaging devices, handheld devices exist for imaging localized areas, but these are limited to specific regions, and precise skin diagnosis of the user is impossible with this limited localized imaging.

[0013] And, just like a device that photographs the entire face, it is impossible to penetrate the dermis layer.

[0014] The present invention aims to provide a mobile skin diagnostic device that enables real-time observation, diagnosis, and prescription of not only the epidermis but also the dermis layer without difficulty in dermatology or aesthetics, thereby reducing deviations in treatment results caused by different experience levels.

[0015] The present invention aims to provide a mobile skin diagnostic device that not only enables doctors to perform precise medical examinations but also significantly reduces the risk of medical accidents for patients.

[0016] The present invention aims to provide a mobile skin diagnostic device that can drastically reduce image registration time and significantly increase precision.

[0017] The present invention aims to provide a mobile skin diagnostic device that can adjust the precision of skin diagnosis according to the resolution of the input image and can be used for skin care and aesthetics specialized for the user through acquired skin diagnostic data.

[0018] A mobile skin diagnostic device according to one embodiment of the present invention comprises: a light source device capable of penetrating from the epidermal layer of the skin to the dermal layer; a shooting unit that acquires an image of the upper part of the dermis and subcutaneous tissue incident through the light source device; an inertial sensor that calculates posture data and position data of the shooting unit by measuring angular velocity or acceleration; and a processor connected to the inertial sensor that calculates and records posture data and position data when the shooting unit photographs the skin, and processes the image captured by the shooting unit by aligning the image; a processing unit that receives image data of the upper part of the dermis and subcutaneous tissue and inertial data from the image acquisition device and corrects them to align the entire image of the face; and a skin diagnostic unit that receives the aligned image data of the entire face from the processing unit and generates and displays diagnostic data suitable for the user.

[0019] Here, the light source device comprises a near-infrared light source capable of outputting short-wavelength near-infrared light in the wavelength range of 820 to 940 nm, a white LED light source in the visible light range, a polarizing source with polarization attached to the white LED light source, and a UV light source having a wavelength of less than 420 nm, wherein the near-infrared light source, the white LED light source, the polarizing source, and the UV light source are arranged at 90° intervals on a plane.

[0020] And, the above-mentioned shooting unit includes an image sensor that captures the user's skin and generates at least two or more image data; and a camera equipped with an autofocus lens that automatically focuses on the user's skin to be captured, and an image sensor that captures the image data obtained through the autofocus lens.

[0021] Additionally, the processing unit includes a data acquisition unit that acquires image data and inertial data from the image acquisition device; a data correction unit that precisely corrects the acquired image data and pose data; and an image matching unit that matches the entire image of the face with the corrected pose data and image data.

[0022] In addition, the data correction unit is equipped with one of a Kalman filter, an extended Kalman filter, or a neural network.

[0023] In addition, the image matching unit uses the pose data when generating the density map of the image.

[0024] In addition, the inertial sensor is equipped with a gyroscope sensor, or is equipped with both the gyroscope sensor and an accelerometer.

[0025] Additionally, the skin diagnosis unit includes: a diagnosis unit that receives image data of the entire face aligned from the processing unit; a host device that receives image data of the entire face aligned from the diagnosis unit; and an artificial intelligence server that receives image data aligned from the host device via the Internet and transmits and displays diagnosis data suitable for the user to the host device through the user's skin diagnosis image.

[0026] According to the present invention, real-time observation, diagnosis, and prescription are possible not only for the epidermis but also for the dermis layer without difficulty in dermatology or aesthetics, thereby reducing variations in treatment results caused by different levels of experience.

[0027] According to the present invention, not only can doctors provide precise medical care, but the risk of medical accidents for patients can also be drastically reduced.

[0028] According to the present invention, image matching time can be drastically reduced and precision can be significantly increased.

[0029] According to the present invention, the precision of skin diagnosis can be adjusted according to the resolution of the input image, and the acquired skin diagnosis data can be used for skin care and aesthetics specialized for the user.

[0030] FIG. 1 is a schematic diagram of a mobile skin diagnostic device according to one embodiment of the present invention.

[0031] FIG. 2 is a schematic diagram of an image acquisition device according to one embodiment of the present invention.

[0032] FIG. 3 is a block diagram of a processing unit according to one embodiment of the present invention.

[0033] FIG. 4 is a schematic diagram of a skin diagnostic unit according to one embodiment of the present invention.

[0034] Figure 5 is a schematic diagram of the light source device of Figure 1.

[0035] FIG. 6 is a graph showing the wavelength of a light source versus the penetration depth of a light source according to one embodiment of the present invention.

[0036] FIG. 7 is a diagram showing the state of irradiating a light source onto skin and capturing an image with a camera according to one embodiment of the present invention.

[0037] FIG. 8 is a diagram illustrating the process for using each light source and acquiring a final image when photographing a local area according to an embodiment of the present invention.

[0038] FIG. 9 is a photograph showing a final image produced by a near-infrared light source according to one embodiment of the present invention.

[0039] FIG. 10 is a photograph showing a final image produced by a white LED light source according to one embodiment of the present invention.

[0040] FIG. 11 is a photograph showing a final image produced by a polarizing source according to one embodiment of the present invention.

[0041] FIG. 12 is a photograph showing a final image produced by a UV light source according to one embodiment of the present invention.

[0042] FIG. 13 is a diagram showing image matching in image matching according to an embodiment of the present invention.

[0043] FIG. 14 is a diagram illustrating the process of determining an ROI area required for image matching from inertial sensor data according to an embodiment of the present invention and performing matching using the same.

[0044] Hereinafter, a mobile skin diagnostic device according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0045] FIG. 1 is a schematic diagram of a mobile skin diagnostic device according to an embodiment of the present invention, and FIG. 2 is a schematic diagram of an image acquisition device according to an embodiment of the present invention. FIG. 3 is a block diagram of a processing unit according to an embodiment of the present invention, and FIG. 4 is a schematic diagram of a skin diagnostic unit according to an embodiment of the present invention.

[0046] Referring to FIGS. 1 to 4, a mobile skin diagnostic device (1000) according to one embodiment of the present invention may include an image acquisition device (100), a processing unit (200), and a skin diagnostic unit (300).

[0047] The image acquisition device (100) acquires an image of the upper part of the dermis and subcutaneous tissue incident through the light source device (110), calculates and records posture data and position data, and processes the captured image by aligning it.

[0048] The processing unit (200) receives image data of the dermis and subcutaneous tissue and inertial data from the image acquisition device (100), corrects them, and aligns the entire image of the face. USB, CVBS, HD-SDI, LVDS, HDMI, WI-FI, Ethernet, etc., can be connected between the interface unit (150) of the image acquisition device (100) and the data acquisition unit (210) of the processing unit (200).

[0049] The skin diagnosis unit (300) receives image data of the entire face aligned from the processing unit (200) and generates and displays diagnostic data suitable for the user.

[0050] Between the interface section (240) of the processing section (200) and the diagnosis section (310) of the skin diagnosis section (300), WI-FI, UWB, Ethernet, USB, RGB output, HDMI, Display Port, WiDi, WiGiG, CVBS, etc. may be connected.

[0051] Referring to FIG. 2, the image acquisition device (100) may include a light source device (110), a shooting unit (120), an inertial sensor (130), a processor (140), and an interface unit (150).

[0052] The light source device (110) can penetrate from the epidermal layer of the skin to the dermal layer. Since the image acquisition environment that the camera (122) must accept is the skin, the image can be made clearer by maintaining a specific brightness or changing the illumination conditions. This is to enable smoother control by the 3A (Auto Exposure, Auto White Balance, Auto Focus) engine of the processor inside the camera (122).

[0053] In addition, the light source device (110) can use different wavelengths of light to identify different skin feature points.

[0054] The imaging unit (120) acquires an image of the upper part of the dermis and subcutaneous tissue incident through the light source device (110).

[0055] Specifically, the shooting unit (120) is equipped with an image sensor (121) and a camera (122). As the camera (122), a single lens camera or a multiple camera equipped with an optical system including at least one lens may be used.

[0056] The image sensor (121) captures the user's skin to generate at least two image data, and the camera (122) is equipped with an autofocus lens (122a) that automatically focuses on the user's skin to be captured, and an image sensor (122b) that captures the image data obtained through the autofocus lens (122a).

[0057] The inertial sensor (130) measures angular velocity or acceleration to calculate attitude data and position data of the camera unit. Here, the inertial sensor (130) is equipped with a gyroscope sensor to measure attitude, or is equipped with a gyroscope sensor and an accelerometer to measure attitude and position.

[0058] The processor (140) is connected to the inertial sensor (130) to calculate and record posture data and position data when the shooting unit (120) photographs the skin, and processes the image captured by the shooting unit (120) by aligning it.

[0059] And the camera and the processor (140) are connected by a data line (131), such as an FPCB, which transmits the captured image, and the data line (131) transmits the image captured by the camera (122) to the processor (140).

[0060] The posture data contains the angle between the user's skin and the image acquisition device (100), and the position data contains the distance, relative position, relative coordinates, etc. of the image acquisition device (100) to the user's skin. That is, the viewpoint of the shooting unit (120) can be inferred.

[0061] Attitude data and position data can be obtained by integrating the measured angular velocity and acceleration values, and the attitude data and position data can be corrected using other measurements obtained by the inertial sensor (130), such as magnetic force from the magnetic field sensor.

[0062] The interface unit (150) transmits the image processed by alignment in the processor (140) to the data acquisition unit (210) of the processing unit (200).

[0063] Referring to FIG. 3, the processing unit (200) may include a data acquisition unit (210), a data correction unit (220), an image matching unit (230), and an interface unit (240).

[0064] The data acquisition unit (210) acquires image data and inertial data from the image acquisition device (100).

[0065] The data correction unit (220) precisely corrects the acquired image data and pose data. Here, the data correction unit (220) is equipped with one of a Kalman filter, an extended Kalman filter, or a neural network.

[0066] The image matching unit (230) matches the entire image of the face with the corrected posture data and image data. Here, the image matching unit (230) uses the posture data when generating the density map of the image.

[0067] The interface unit (240) transmits the image matched in the image matching unit (230) to the diagnosis unit (310) of the skin diagnosis unit (300).

[0068] Referring to FIG. 4, the skin diagnosis unit (300) may include a diagnosis unit (310), a host device (320), and an artificial intelligence server (330).

[0069] The diagnostic unit (310) receives image data of the entire face aligned from the processing unit (200).

[0070] The host device (320) receives image data of the entire face aligned from the diagnostic unit (310). Here, the host device (320) can communicate with the artificial intelligence server (330) via a smartphone app or a web browser.

[0071] The artificial intelligence server (330) receives image data matched from the host device (320) via the internet and transmits and displays diagnostic data suitable for the user to the host device (320) through the user's skin diagnostic image. At this time, the artificial intelligence server (330) uses an artificial intelligence algorithm such as deep learning to enable a more precise diagnosis through input data from multiple users.

[0072] Based on the diagnosis, we can provide all skincare-related products tailored to the customer, such as cosmetics, skincare products, galvanic devices, and LED masks.

[0073]

[0074] FIG. 5 is a schematic diagram of the light source device of FIG. 1, and FIG. 6 is a graph showing the wavelength of the light source versus the penetration depth of the light source according to one embodiment of the present invention.

[0075] Referring to FIG. 5, the light source device (110) may include a near-infrared light source (111), a white LED light source (112), a polarizing source (113), and a UV light source (114). Here, the near-infrared light source (111), the white LED light source (112), the polarizing source (113), and the UV light source (114) are arranged at 90° intervals on a plane.

[0076] The near-infrared light source (111) is capable of outputting short-wavelength near-infrared light in the wavelength range of 820 to 940 nm. In one embodiment of the present invention, in addition to the light sources used for conventional skin diagnosis, there must be a light source capable of penetrating from the epidermis to the dermis layer of 4 to 5 mm, so a 940 nm short-wavelength near-infrared light source (111) capable of penetrating to the dermis layer is used as shown in FIG. 6.

[0077] In Figure 6, it can be seen that the skin consists of the epidermis, including the stratum corneum, the dermis, and the hypodermis.

[0078] Near-infrared light source (111) of 940 nm or higher can reach the dermis layer of the inner layer of the skin and is reflected and refracted when it encounters subcutaneous tissue and blood vessels.

[0079] In one embodiment of the present invention, as shown in FIG. 7, the reflected light of the corresponding light source is extracted and imaged using a single lens camera or multiple cameras, and then the time from when the near-infrared light source (111) is incident until the reflected light source is extracted by the camera is measured and reconstructed in three dimensions. This method is widely used in existing methods for acquiring three-dimensional images and is commonly referred to as Time-of-Flight (ToF).

[0080] In one embodiment of the present invention, a structured light method is proposed to enable a three-dimensional configuration other than the ToF method, and this takes the form of changing the irradiated pattern of the near-infrared light source (111) (in addition to the ToF method, the structured light method can be selectively applied).

[0081] The white LED light source (112) is in the visible light range and can be used to accurately measure the condition of the skin.

[0082] The polarizing source (113) is a white LED light source (112) with polarization attached, and can capture the depth and texture of the wrinkles on the skin.

[0083] The UV light source (114) has a wavelength of less than 420 nm and can detect porphyrin and pigmentation on the skin surface or under the epidermis.

[0084]

[0085] FIG. 7 is a diagram showing the state of irradiating a light source onto skin and capturing an image with a camera according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating the use of each light source and the process for acquiring a final image when photographing a local area according to an embodiment of the present invention. FIG. 9 is a photograph showing the final image by a near-infrared light source according to an embodiment of the present invention, and FIG. 10 is a photograph showing the final image by a white LED light source according to an embodiment of the present invention. FIG. 11 is a photograph showing the final image by a polarizing light source according to an embodiment of the present invention, and FIG. 12 is a photograph showing the final image by a UV light source according to an embodiment of the present invention.

[0086] Referring to Fig. 7, since the entire skin condition, including the dermis layer, must be assessed in a single scan, each light source (111, 112, 113, 114) must be irradiated at least once per target scene. To achieve this, the camera must be capable of ultra-high-speed imaging and must have an imaging speed of at least 180fps.

[0087] In each scene, a local area is filmed; at this time, each light source is irradiated sequentially on the same scene to acquire images of the same local area for each light source.

[0088] This process is illustrated in detail in FIG. 8, and images captured by each light source (111, 112, 113, 114) can be obtained through a final image matching process to obtain images of the skin condition of the entire face for each light source as shown in FIG. 9 to 12.

[0089]

[0090] FIG. 13 is a diagram showing image matching in image matching according to an embodiment of the present invention, and FIG. 14 is a diagram showing the process of determining a Region of Interest (ROI) area required for image matching from inertial sensor data according to an embodiment of the present invention and performing matching using the ROI.

[0091] Referring to FIG. 13, a handheld skin imaging device is used to scan the face while being held in one hand for skin imaging. When the user scans the entire face by rotating the skin imaging device in a full circle, an image of the entire face is acquired through image registration.

[0092] In order to connect local areas captured at once, a process of matching two-dimensional images from consecutive frames is required as shown in Fig. 12, and image matching algorithms such as SIFT, SURF, FERN, and Optical Flow are used.

[0093] Image matching algorithms locate key points in images to determine the position of the same subject across two different images; by applying this to sequential camera inputs, it becomes possible to identify the location of a specific key point within a series of focused consecutive image frames. In this process, position and pose data from inertial sensors are used together for fast image matching.

[0094] Focused continuous images generally have HD resolution (1280×720), and can have higher resolution to increase precision.

[0095] To use image matching in two consecutive image frames, a method of finding feature points across the entire image area and matching them is typically used. However, this presents a problem in that it cannot be processed in real-time on a small embedded CPU. In other words, performing the operation to find feature points across the entire frame has the disadvantage of being very difficult to process in real-time on an embedded device.

[0096] To solve this, one embodiment of the present invention presents a method in which, as shown in FIG. 14, an Inertial Navigation System (INU) is additionally used to find only the overlapping parts in consecutive frames and to perform image matching operations only on those parts. Furthermore, the present invention may additionally use an Inertial Measurement Unit (IMU).

[0097] Through the processes of inertial sensor sensing and image stitching, the current position and attitude of the camera can be predicted, and a stitched image can be generated for the focused continuous image.

[0098]

[0099] Referring to FIG. 14, the image acquisition device moves from the location (P1) where the first image data was captured to the second location (P2) where the second image data was captured.

[0100] Information regarding the displacement (distance moved) of the imaging unit and the angle between the imaging unit and the user's skin, that is, the viewpoint of the imaging unit, can be calculated by analyzing the posture and position data of the first image data and the second image data, respectively, to determine the degree of change.

[0101] In other words, if the image acquisition device moves within a preset shooting area and captures the user's skin, there must be an area where the first image data and the second image data overlap. Accordingly, the processor searches for the area where the two image data overlap as a Region of Interest (ROI) and then detects feature points within that area.

[0102] Feature detection and image matching in the processor utilize existing algorithms such as SIFT (Scale Invariant Feature), SURF (Speeded Up Robust Features), FERN, HOG (Histogram of Oriented Gradient), and Optical Flow.

[0103] Generally, the registration of two image data involves extracting feature points from the entire image and then searching for common regions in the segmented image data based on the extracted feature points.

[0104] Continuous image data, or a panoramic image, is generated through a process of aligning images based on the searched areas.

[0105] However, since extracting feature points from the entire area is a computationally intensive operation, there is a problem in that real-time processing is difficult on small embedded devices.

[0106] Accordingly, unlike general methods, image matching is performed by first finding a region of interest, then deriving feature points within the region of interest, and matching two or more image data by determining whether each feature point matches.

[0107] As disclosed in Korean Patent Publication No. 10-2023-0095716, the image matching portion comprises feature points (u11, u12, u) of an image in two or more focused consecutive images. 13 , u 21 , u 22 , u 23 , u 24If three or more common feature points (x1, x2, x3) are found, the virtual viewpoint (m1, m2) of the camera can be inferred using the location of the feature points.

[0108] In this process, the inferred camera viewpoint or matching point may produce outliers due to incorrect feature point matching, which are filtered using RANSAC (Random Sample Consensus).

[0109] Noise removal is performed on the completed points, which contain high-frequency noise components, by utilizing the alignment formed by the points that have completed filtering (using the formed alignment points).

[0110] In one embodiment of the present invention, noise removal was performed using a bilateral filter considering speed and performance, but is not limited thereto.

[0111]

[0112] The embodiments of the present invention described above have been explained with reference to the embodiments illustrated in the drawings for the sake of understanding, but this is merely illustrative and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom.

[0113] Therefore, the true technical scope of protection of the present invention should be determined by the appended claims.

Claims

1. An image acquisition device comprising: a light source device capable of penetrating from the epidermal layer of the skin to the dermal layer; a shooting unit for acquiring an image of the upper part of the dermis and subcutaneous tissue incident through the light source device; an inertial sensor for calculating attitude data and position data of the shooting unit by measuring angular velocity or acceleration; and a processor connected to the inertial sensor for calculating and recording attitude data and position data when the shooting unit photographs the skin, and for aligning and processing the image captured by the shooting unit; A processing unit that receives image data and inertial data of the upper dermis and subcutaneous tissue from the image acquisition device and corrects them to align the entire image of the face; and A mobile skin diagnostic device characterized by including a skin diagnostic unit that receives image data of the entire face aligned from the processing unit and generates and displays diagnostic data suitable for the user.

2. In Paragraph 1, The above light source device comprises a near-infrared light source capable of outputting short-wavelength near-infrared light in the wavelength range of 820 to 940 nm, a white LED light source in the visible light range, a polarizing source having polarization attached to the white LED light source, and a UV light source having a wavelength of less than 420 nm, wherein the near-infrared light source, the white LED light source, the polarizing source, and the UV light source are arranged at 90° intervals on a plane, characterized in that they are a mobile skin diagnostic device.

3. In Paragraph 1, The above-mentioned imaging unit is, An image sensor that captures the user's skin and generates at least two or more image data; and A mobile skin diagnostic device characterized by including a camera equipped with an autofocus lens that automatically focuses on the skin of a user being photographed, and an image sensor that captures image data acquired through the autofocus lens.

4. In Paragraph 3, The above processing unit is, A data acquisition unit that acquires image data and inertial data from the above image acquisition device; A data correction unit that precisely corrects acquired image data and pose data; and A mobile skin diagnostic device characterized by including an image matching unit that matches the entire image of the face with corrected posture data and image data.

5. In Paragraph 4, A mobile skin diagnostic device characterized by the above data correction unit being equipped with one of a Kalman filter, an extended Kalman filter, or a neural network.

6. In Paragraph 4, A mobile skin diagnostic device characterized by the above image matching unit using the above posture data when generating a density map of the above image.

7. In Paragraph 1, A mobile skin diagnostic device characterized in that the above-mentioned inertial sensor is equipped with a gyroscope sensor or is equipped with both a gyroscope sensor and an accelerometer.

8. In Paragraph 1, The above-mentioned skin diagnostic unit is, A diagnostic unit that receives image data of the entire face aligned from the processing unit above; A host device that receives image data of the entire face aligned from the diagnostic unit; and A mobile skin diagnosis device characterized by including an artificial intelligence server that receives image data matched from the host device via the internet and transmits and displays diagnostic data suitable for the user to the host device through the user's skin diagnosis image.

Images