System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method

Abstract

The invention relates to an artificial intelligence-aided system based on deep learning methods for determining botox injection points in the upper facial region more precisely and accurately within the field of cosmetic dermatology.

Description

[0001] DESCRIPTION

[0002] SYSTEM FOR ARTIFICIAL INTELLIGENCE-AIDED AUTOMATIC DETECTION OF BOTOX INJECTION POINTS IN COSMETIC DERMATOLOGY WITH DEEP LEARNING METHOD

[0003] Technical Field of the Invention

[0004] The invention relates to an artificial intelligence-aided system based on deep learning methods for determining botox injection points in the upper facial region more precisely and accurately within the field of cosmetic dermatology.

[0005] State of the Art

[0006] Today, botox applications hold a significant place in the field of aesthetics; however, the determination of correct injection points is crucial for the quality of the procedure. To determine the correct injection points, a specialist dermatologist or aesthetic surgeon must carefully assess the regions where wrinkles and expression lines are present. This process requires great attention, is time-consuming and tiring, and may lead to variations among different specialists and within the same specialist. Furthermore, human facial expressions are generated by 43 muscles, and these muscles can produce over 100,000 different variations. Therefore, it is not sufficient for the injector merely to be a specialist, they must also possess a highly detailed knowledge of facial anatomy.

[0007] In a botox injection point determination system, it is of great importance to identify injection points with correct timing and positioning to avoid detecting incorrect injection points and to prevent negative effects on the quality of the application. Injection points that are not determined correctly can lead to asymmetric results and insufficient effect, especially when applied in areas that are not suitable for facial muscle structure.

[0008] In the methods in the state of the art, computer-aided diagnostic systems in cosmetic dermatology focus more on visual changes on the skin surface, such as "automatic diagnosis of skin blemishes and pigmentation", "analysis of wrinkles and signs of aging", and "acne and pustule diagnosis". Such systems are focused on assessing skin structure and identifying specific skin problems such as aging, blemishes or acne. Existing systems do not provide support for accurate and repeatable identification of injection points, and deep learning-based solutions are needed to address the shortcomings in this area.

[0009] Manual determination of injection points is a process that requires expertise and carries a risk of error. Therefore, automatic detection of Botox injection points using deep learning methods can both improve accuracy and ease the workload of specialists. Such an artificial intelligence-aided application aims to ensure repeatability, standardize procedures, and increase safety, thereby increasing patient satisfaction and optimizing procedure time.

[0010] Summary and Objects of the Invention

[0011] The invention relates to an artificial intelligence-aided system based on deep learning methods for determining botox injection points in the upper facial region more precisely and accurately within the field of cosmetic dermatology.

[0012] An object of the invention is to enable precise and repeatable diagnostic capability. Injection points are automatically determined, ensuring high precision and sensitivity. In this way, Botox procedures are applied at the right points and repeatable results are obtained without differences occurring between specialists.

[0013] Another object of the invention is to enable fast and efficient processing. The automated system is faster than manually identifying points, which reduces processing time. With the increased efficiency, more patients can be served in a shorter time.

[0014] Another object of the invention is to reduce the risk of human error, enabling accurate detection of injection points. This helps to apply Botox effectively in the desired muscle groups, reducing side effects.

[0015] Another object of the invention is to enable a wider range of healthcare personnel to perform the procedure by lowering the level of expertise required for the people performing Botox procedures. This system can also be used to support the training of practitioners. Another object of the invention is to minimize the trial-and-error process and make the process more precise. This results in less pain and discomfort for patients.

[0016] Description of the Drawings

[0017] Fig. 1. Drawing showing a schematic view of the system of the invention.

[0018] Description of the References in the Drawings

[0019] 1. Dataset unit

[0020] 2. Training Unit

[0021] 3. CAD diagnostic unit

[0022] Detailed Description of the Invention

[0023] The invention relates to an artificial intelligence-aided system based on deep learning methods for determining botox injection points in the upper facial region more precisely and accurately within the field of cosmetic dermatology.

[0024] The dataset unit (1) consists of the collection, pre-processing and labeling of images suitable for the detection of injection points. Images obtained with different facial expressions are important for creating a high-quality and representative dataset for training the model.

[0025] The Training Unit (2) comprises the implementation of the YOLOv8 object detection algorithm and the optimization of anchor box dimensions using the K-means clustering algorithm. Through hyperparameter tuning, it enhances model accuracy, enabling more effective detection of injection points.

[0026] The CAD (Computer-Aided Design) Diagnostic Unit (3) ensures the evaluation of the model obtained during the training phase on an independent test dataset. Based on the test results, the model’s generalization capability and performance are analyzed, aiming to achieve the level of accuracy and speed required to guide healthcare professionals.

[0027] The first step applied in the dataset unit (1) is the collection of suitable images. In this process, images of each subject with five different facial expressions (neutral expression, frown, raised eyebrow, right smile and left smile) are captured with the camera. These expressions help to analyze the role of muscle movements in determining injection points. In the second step, the acquired images are pre- processed in order for the model to learn correctly. In the pre-processing stage, all images are first resized to match the model input size. Geometric rigid data augmentation techniques are then used to expand the dataset and increase the generalization ability of the model. Finally, the images are normalized using ImageNet color statistics. This normalization process scales the pixel values to these means and standard deviations for the respective RGB channels, making the model training process more stable and efficient. Finally, the images are labeled by a specialist dermatologist. The labeled images are saved in JPG format and the labeled injection location information is saved in a txt file. This data is then used for training and validation of the model.

[0028] The training unit (2) runs on the server and uses the YOLOv8 object detection algorithm to detect injection points. Secondly, the K-means clustering algorithm is used to optimize the dimensions of the problem-specific anchor boxes. Identifying suitable anchor boxes improves the performance of the model by increasing accuracy in object detection tasks. This process allows the model to achieve better overall results and more accurate identification of Botox injection points. As is well known, hyperparameter tuning in deep learning models is a critical process that significantly affects the performance of the model. Hyperparameters are tuning parameters that control the model's learning process and optimize results; therefore, correct hyperparameter tuning increases accuracy, speed, and overall efficiency in object detection tasks. The YOLOv8 algorithm and the artificial bee colony optimization algorithm were used together for optimal hyperparameter tuning. At this stage, the learning rate, momentum, weight reduction, and number of epochs parameters were adjusted to give the best performance measurement metrics (Precision, Sensitivity, f1 -score) on the validation set. The model obtained as a result of training is saved to be used in the test phase. This saving process ensures that all the parameters needed to evaluate the model's performance and to be used in future applications are preserved.

[0029] The CAD (computer-aided design) diagnostic unit (3) runs on a server, a test dataset is prepared independently of the training and the model obtained during the training phase is evaluated on the test dataset. The test unit is important for understanding the generalization ability of the model and is evaluated by four metrics: accuracy, precision, sensitivity, and F1 -score. Accuracy indicates the rate at which the model correctly identifies injection points, precision indicates the rate at which the identified injection points are correct, and sensitivity indicates how many of the actual injection points were correctly identified. The F1 -score combines the precision and sensitivity metrics of the model, providing a balanced assessment of both metrics. According to the test results, the performance of the model is evaluated and necessary improvements are made. At this stage, the model aims to reach a level of accuracy and speed that can guide health professionals. The model that successfully passes the test phase is integrated as the final system. The integrated system is able to detect botox injection points through live images taken with the camera and provides guidance to professionals.

[0030] The operation method of the invention is as follows;

[0031] - Capturing facial expression images of subjects with a camera through the dataset unit (1 ),

[0032] - Performing data pre-processing (data augmentation, normalization) on the images via a server through the dataset unit (1 ),

[0033] - Labeling the images through an interface by a specialist using the dataset unit (1),

[0034] - Creating the deep learning model (YOLOv8) on the server through the training unit (2),

[0035] - Determining problem-specific anchor box sizes using the K-means clustering algorithm on the server through the training unit (2),

[0036] - Training the model on the server through the training unit (2),

[0037] - Performing hyperparameter optimization using the artificial bee colony optimization algorithm on the server through the training unit (2),

[0038] - Saving the trained model on the server through the training unit (2), - Evaluating the model obtained from the training unit (2) on the server on a test dataset prepared independently of the training through the CAD Diagnostic Unit (3), using accuracy, precision, sensitivity, and F1 -score metrics,

[0039] - Assessing the model performance based on test results on the server through the CAD diagnostic unit (3),

[0040] - Integrating the successfully passed model as a final system on the server through the CAD diagnostic unit (3).

[0041] In the process step of capturing facial expression images of subjects with a camera through the dataset unit (1 ), facial expressions are captured in five different facial expressions: neutral expression, frown, raised eyebrow, side-right smile and side-left smile.

[0042] In the process step of preparing a test dataset independent of the training on the server and evaluating the trained model on the test dataset through the CAD diagnostics unit (3); accuracy indicates the rate at which the model correctly identifies injection points, precision indicates the rate at which the identified injection points are correct, sensitivity indicates how many of the actual injection points were correctly identified, the F1 - score combines the precision and sensitivity metrics of the model, giving the results of a balanced assessment of both metrics.

[0043] The preferred mode of operation of the invention is as follows;

[0044] Clinical images will be collected in the dataset unit (1 ) from patients who apply for Botox treatment and who do not have any contraindications for Botox treatment (age, pregnancy, breastfeeding, skin disease) for the purpose of preparing the working database. Images will be captured using a camera with standard settings. Five different facial expressions will be captured from each patient: neutral expression, frown, raised eyebrows, smile (right side), and smile (left side), and these images will focus on the upper face region. If noise, blurring, etc. occur in images obtained with imaging devices due to device-related or lighting conditions, the performance of the methods will be improved by applying pre-processing such as noise reduction and image enhancement. In addition, the following physical elements will be removed prior to the imaging process. This step is one of the measures taken to ensure that the resulting images are analyzed with the highest quality and with the least possible error. This ensures optimal conditions for accurate and reliable detection of Botox injection points.

[0045] • Make-up and Skin Products: Heavy makeup or skin products can make an accurate detection difficult. For this reason, the patient is asked to remove any makeup or similar products, and it is ensured that they are completely removed.

[0046] • Hair: Hair covering the face or strands of hair falling onto the facial area may prevent imaging of Botox points. For this reason, the hair is gathered at the back to prevent it from falling on the facial region.

[0047] • Accessories: Piercings or glasses on the face may obstruct accurate imaging. The patient is therefore asked to remove any piercings and glasses.

[0048] If the number of images is insufficient for training the proposed deep learning models, data augmentation techniques, which are widely used in such cases, will be applied. This method aims to artificially expand the size of the training dataset and enhance the model’s generalization ability by creating modified versions of images through geometric rigid transformations, such as random rotation, adding reflections, and minor zooming, thereby achieving more robust and reliable outcomes. Subsequently, the images will be normalized using ImageNet color statistics: RGB mean = [0.485, 0.456, 0.406] and standard deviation = [0.229, 0.224, 0.225]. This normalization process scales the pixel values to these means and standard deviations for the respective RGB channels, making the model training process more stable and efficient.

[0049] Finally, the dataset will be labeled by a specialist dermatologist. Injection regions of interest will be labeled with rectangular label type. The bounding box drawn by the user with the rectangular label type is a vector in [x y width height] format, and the components of this vector consist of the horizontal and vertical coordinates of the upper left corner of the injection region of interest, the width and height of the injection region of interest values, respectively. The tagged images will be saved in JSON format and then converted to the format required by the algorithms (xml, txt, etc.) for training and validation of the algorithms. The YOLOv8 model architecture created in the Training Unit (2) consists of Backbone, Neck and Head sections. The backbone part of the model uses the Cross Stage Partial (CSP) architecture to split the feature map into two parts. The first part uses convolution operations, while the second part is combined with the output of the previous part. With the CSP architecture, the computational cost of the model is reduced while increasing the learning ability of CNNs. YOLOv8 differs from YOLO v5 by combining the C3 module with the ELAN concept from YOLOv7 to introduce the C2f module, this module enables the model to provide richer gradient flow information. The C3 module consists of 3 ConvModules and n DarknetBottleNecks, while the C2f module consists of 2 ConvModules and n DarknetBottleNecks connected to each other by Split and Concat. ConvModule consists of Conv-BN-SiLU layers, and n refers to the number of bottlenecks. Furthermore, the number of blocks in each stage has been reduced compared to YOLOv5 to further reduce the computational cost. The model has 3, 6, 6, 3 blocks from Stage 1 to Stage 4 respectively. In addition, the Spatial Pyramid Pooling - Fast (SPPF) module, which is an improved form of the Spatial Pyramid Pooling (SPP) module, was used in Phase 4. This module aims to increase the detection speed of the model. All these changes have resulted in a better learning ability of the model and a shorter detection time.

[0050] As the depth of network architectures increases, up to a certain level, more feature information can be obtained, leading to better predictions. On the other hand, extremely deep networks reduce the location information of the object and too much convolution operations can result in information loss for small objects. Therefore, it is necessary to use Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) architectures for multi-scale feature aggregation. In the Neck part of the model architecture, multiscale feature aggregation is used, combining features from different layers of the network. The upper layers obtain more information due to additional network layers, while the lower layers retain location information as they have fewer convolution layers. Inspired by YOLOv5, FPN performs bottom-up resampling to enhance feature information in lower feature maps, while PAN performs top-down resampling to incorporate more information from higher feature maps. These two feature outputs are then combined to produce precise predictions for images of varying sizes. In the model architecture, an FP-PAN (Feature Pyramid-Path Aggregation Network) is utilized, and convolution operations in the resampling process are removed to reduce computational cost. Unlike YOLOv5 model, the classification and detection heads are used separately with a decoupled head structure. The "Anchor-Base" method is used to determine four offsets of the object from the reference anchors using several reference anchors in the image. This method adjusts the object’s exact location based on corresponding anchors and offsets. In contrast, YOLOv8 adopts an "Anchor- Free" method. This method determines the center of the object and estimates the distance between the center and the bounding box. That is, it tries to determine the bounding box of the object directly without using reference anchors.

[0051] Instead of using existing anchor boxes, the YOLOv8 method will use the K-means algorithm to determine the dimensions of the anchor boxes problem-specific to the searched object. This implementation enables better results by improving the model's accuracy in object detection tasks. This process makes a unique contribution by improving both the performance and the effectiveness of the model in practice. Moreover, the use of the artificial bee colony optimization algorithm for hyperparameter tuning makes the training process of the model more efficient and effective, while increasing the reliability of the results.

[0052] After the CAD Diagnostics (Test) Unit (3) training is completed, a test dataset independent of the training will be prepared for the CAD diagnostics (test) stage. This is a critical step in understanding the model's generalization ability and is a method often overlooked in the existing literature. This enables a robust assessment of how effective the model would be in real-world scenarios. The trained model will be reloaded and applied to the test set. The evaluation with four metrics such as accuracy, precision, sensitivity, and F1 -score will provide an opportunity to comprehensively analyze the performance of the model. Using these metrics in combination increases the model's accuracy and reliability in object detection, paving the way for the creation of a system for healthcare professionals.

[0053] The goal of the model to reach a level of accuracy and speed that can guide health professionals emphasizes the practical applicability of the study and its contribution to the medical field. The integration of the model, which has successfully passed the test stage, into the final system will provide an important innovation by increasing automation and accuracy in the detection of Botox injection points, while enabling the results obtained to be transferred to clinical applications. These features enhance the uniqueness of the study and its potential impact in the field of health.

Claims

CLAIMS1. System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method, characterized in that it comprises:- at least one dataset unit (1) that captures different facial expression images of the subjects with a camera, subjects the images obtained to pre-processing on a server, and allows the images to be labeled by a specialist through an interface,- at least one training unit (2) running on the server, which optimizes the dimensions of the anchor boxes by implementing the object detection algorithm and using the K-means clustering algorithm, optimizes the hyperparameters with the artificial bee colony optimization algorithm, and trains and saves the deep learning model,- CAD diagnostic unit (3) running on the server, which evaluates the model obtained from the training unit (2) with accuracy, precision, sensitivity, and F1 -score metrics on a test dataset prepared independently of the training, evaluates the performance of the model according to the test results and makes improvements, and integrates the model that successfully passes the test phase as a final system.

2. Method for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method, characterized in that it comprises the process steps of:- Capturing facial expression images of subjects with a camera through the dataset unit (1 ),- Performing data pre-processing on the images via a server through the dataset unit (1),- Labeling the images through an interface by a specialist using the dataset unit (1),- Creating the deep learning model on the server through the training unit (2),- Determining problem-specific anchor box sizes using the K-means clustering algorithm on the server through the training unit (2),- Training the model on the server through the training unit (2),- Performing hyperparameter optimization using the artificial bee colony optimization algorithm on the server through the training unit (2),- Saving the trained model on the server through the training unit (2),- Evaluating the model obtained from the training unit (2) on the server on a test dataset prepared independently of the training through the CAD diagnostic unit (3), using accuracy, precision, sensitivity, and F1 -score metrics,- Assessing the model performance based on test results on the server through the CAD diagnostic unit (3),- Integrating the successfully passed model as a final system on the server through the CAD diagnostic unit (3).

3. System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method according to claim 1 , characterized in that it comprises a dataset unit (1 ) that captures images of subjects with neutral expression, frown, raised eyebrows, right smile, and left smile with a camera.

4. System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method according to claim 1 , characterized in that it comprises a dataset unit (1) that resizes the acquired images in accordance with the model input size with the server, uses geometric rigid data augmentation techniques to expand the dataset and enhance the generalization ability of the model, and normalizes it using color statistics.

5. System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method according to claim 1 , characterized in that it comprises a data set unit (1 ) that allows the dermatologist to label injection regions of interest with a rectangular label type via an interface.

6. System for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method according to claim1 , characterized in that it comprises a CAD diagnostic unit (3) which calculates the accuracy metric that detects the rate at which the model correctly identifies injection points, the precision metric that detects the rate at which the identified injection points are correct, the sensitivity metric that detects how many of the actual injection points were correctly identified, and the F1 -score metric that combines the precision and sensitivity metrics of the model, giving the results of a balanced assessment of both metrics.

7. Method for artificial intelligence-aided automatic detection of botox injection points in cosmetic dermatology with deep learning method according to claim2, characterized in that in the process step of capturing facial expression images of the subjects with the camera through the dataset unit (1), the facial expressions are: neutral expression, frown, raised eyebrows, side-right smile and side-left smile.

Images