System for providing artificial intelligence (AI)-based personalized cosmetic recommendation service

Abstract

Provided is a system for providing an AI-based personalized cosmetic recommendation service, the system comprising: a user terminal for photographing a face and receiving recommendations for cosmetics; and a recommendation service providing server including a reception unit for receiving a face image from the user terminal, a de-identification unit for storing and managing data by dividing the face in the face image into at least one skin zone, an analysis unit for analyzing the skin of the at least one skin zone using a pre-constructed analysis model, and a recommendation unit for extracting and recommending cosmetics corresponding to analysis results of the analysis model.

Description

AI-based personalized cosmetics recommendation service provision system

[0001] The present invention relates to an AI-based personalized cosmetic recommendation service provision system, and provides a system capable of providing personalized cosmetic recommendations as well as protecting personal information by dividing a face image into skin zones for analysis, storage, and management.

[0002] The rapid advancement of artificial intelligence technology is driving innovation in the manufacturing sector. In particular, regarding personalized skincare products, while humans previously analyzed skin conditions and manufactured customized items, systems can now be established that utilize image analysis and AI algorithms to analyze a user's skin condition in real time and automatically combine optimal cosmetics based on this analysis. The introduction of such technology enables manufacturers to improve productivity while simultaneously 혁신적으로 enhancing the user experience, and allows users to select perfectly customized cosmetics through accurate analysis of their skin. Furthermore, automation powered by AI offers opportunities to minimize errors in the manufacturing process and maximize efficiency.

[0003] At this time, a method for recommending cosmetics by analyzing skin images was researched and developed. In this regard, prior art Korean Published Patent No. 2023-0018321 (published February 7, 2023) and Korean Published Patent No. 2022-0156126 (published November 25, 2022) respectively disclose a configuration for extracting the skin type with the highest similarity by comparing a skin type collected from a user terminal with a previously stored skin type and recommending cosmetics corresponding to the extracted skin type, and a configuration for taking a skin image for skin diagnosis, diagnosing the skin by analyzing the skin image, and then recommending cosmetics based on the skin diagnosis information.

[0004] However, in the former case, cosmetics are recommended based on skin similar to the user rather than the user's actual skin, so strictly speaking, it is not a personalized recommendation. In the latter case, since the entire facial image is used to analyze skin images, the facial image capable of identifying an individual remains intact, posing a risk of personal information leakage. Furthermore, because the necessary ingredients and functions vary by area of ​​the face, it is difficult to provide customized solutions. Therefore, research and development of a system capable of analyzing and recommending customized products tailored to a user's skin condition without the risk of personal information leakage is required.

[0005] One embodiment of the present invention provides an AI-based personalized cosmetic recommendation service system capable of de-identifying an individual by receiving a face image from a user terminal, processing the face image by dividing it into at least one skin zone for storage and management, analyzing the skin of at least one skin zone using an analysis model, and extracting a mask pack corresponding to the analysis result to extract and recommend a mask pack customized to the individual. However, the technical problem to be solved by this embodiment is not limited to the technical problem described above, and other technical problems may exist.

[0006] As a technical means for achieving the aforementioned technical problem, one embodiment of the present invention includes a recommendation service providing server comprising: a user terminal that photographs a face and receives recommendations for cosmetics; a receiving unit that receives a face image from the user terminal; a de-identification unit that divides the face of the face image into at least one skin zone and stores and manages data; an analysis unit that analyzes the skin of at least one skin zone using a pre-established analysis model; and a recommendation unit that extracts and recommends cosmetics corresponding to the analysis results of the analysis model.

[0007] According to any one of the means for solving the problem of the present invention described above, when a face image is received from a user terminal, the face image is processed to de-identify an individual by dividing it into at least one skin zone, storing and managing it, and after analyzing the skin of at least one skin zone using an analysis model, a mask pack corresponding to the analysis result is extracted, thereby enabling the extraction and recommendation of a mask pack customized to the individual. Furthermore, by securing data for each body part, it is possible to produce customized cosmetics and patches for each body part, and to provide not only customized solutions for the entire face but also specialized and segmented customized care services for each body part.

[0008] Furthermore, removing the eyes and mouth, which do not belong to the skin, from the segmented face image improves skin measurement accuracy, and the reduction in measurement data speeds up processing. In other words, both the accuracy of skin measurement and processing speed are enhanced.

[0009] FIG. 1 is a diagram illustrating an AI-based personalized cosmetic recommendation service provision system according to one embodiment of the present invention.

[0010] Figure 2 is a block diagram illustrating a recommendation service provider server included in the system of Figure 1.

[0011] FIG. 3 is a diagram illustrating an embodiment in which an AI-based personalized cosmetic recommendation service according to an embodiment of the present invention is implemented.

[0012] FIG. 4 is a flowchart illustrating a method for providing an AI-based personalized cosmetic recommendation service according to an embodiment of the present invention.

[0013] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0014] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components, and it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0015] Terms such as "approximately" and "substantially" as used throughout the specification are used to mean at or near the stated value when inherent manufacturing and material tolerances are presented in the said meaning, and are used to prevent unscrupulous infringers from unfairly exploiting the disclosure in which precise or absolute values ​​are mentioned to aid in understanding the invention. Terms such as "step of" or "step of" as used throughout the specification of the invention do not mean "step for."

[0016] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized by one piece of hardware. Meanwhile, "part" is not limited to software or hardware, and "part" may be configured to reside in an addressable storage medium or configured to run on one or more processors. Accordingly, as an example, "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0017] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0018] In this specification, some of the operations or functions described as mapping or matching with a terminal may be interpreted as meaning mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data.

[0019] In this specification, the term "skin zone" may be an area formed by at least one part of the face or a combination of at least one part. Accordingly, since it includes a single part, it is not necessary for two or more parts to be combined; therefore, it is defined as a concept that includes single parts such as the eyes, nose, mouth, forehead, cheeks, chin, and neck.

[0020] The present invention will be described in detail below with reference to the attached drawings.

[0021] FIG. 1 is a diagram illustrating an AI-based personalized cosmetic recommendation service provision system according to an embodiment of the present invention. Referring to FIG. 1, the AI-based personalized cosmetic recommendation service provision system (1) may include at least one user terminal (100), a recommendation service provision server (300), and at least one manufacturing device (400). However, since the AI-based personalized cosmetic recommendation service provision system (1) of FIG. 1 is merely an embodiment of the present invention, the present invention is not to be interpreted as being limited by FIG. 1.

[0022] At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one user terminal (100) can be connected to a recommendation service provider server (300) through the network (200). And, the recommendation service provider server (300) can be connected to at least one user terminal (100) and at least one manufacturing device (400) through the network (200). Also, at least one manufacturing device (400) can be connected to the recommendation service provider server (300) through the network (200).

[0023] Here, a network refers to a connection structure capable of exchanging information among individual nodes, such as multiple terminals and servers. Examples of such networks include Local Area Networks (LANs), Wide Area Networks (WANs), the World Wide Web (WWW), wired and wireless data networks, telephone networks, and wired and wireless television networks. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), 5G NR (New Radio), 6G (6th Generation of Cellular Networks), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc.

[0024] In the following, the term "at least one" is defined as a term including both singular and plural forms, and it will be obvious that even if the term "at least one" does not exist, each component may exist in a singular or plural form and may mean singular or plural. Furthermore, whether each component is provided in a singular or plural form may be changed according to the embodiment.

[0025] At least one user terminal (100) may be a terminal of a user who receives a recommendation for a mask pack by transmitting a face image to a recommendation service providing server (300) using a web page, app page, program, or application related to an AI-based personalized cosmetic recommendation service.

[0026] Here, at least one user terminal (100) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop equipped with a web browser, a desktop, a laptop, etc. At this time, at least one user terminal (100) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one user terminal (100) may include all kinds of handheld-based wireless communication devices, such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, etc.

[0027] The recommendation service providing server (300) may be a server that provides an AI-based personalized cosmetic recommendation service webpage, app page, program, or application. Additionally, the recommendation service providing server (300) may be a server that receives a face image from a user terminal (100), divides the face image by skin zone, inputs the divided face images by skin zone into a pre-established analysis model to output a skin condition, and extracts and recommends a mask pack corresponding to the skin condition.

[0028] Here, the recommendation service providing server (300) can be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop, a desktop, a laptop equipped with a web browser.

[0029] At least one manufacturing device (400) may be a device that manufactures a mask pack by mixing materials according to the ingredients and ratios of the mask pack recommended by the recommendation service providing server (300) when a manufacturing request is made from a user terminal (100) using a web page, app page, program, or application related to an AI-based personalized cosmetic recommendation service.

[0030] Here, at least one manufacturing device (400) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a navigation system, a laptop equipped with a web browser, a desktop, a laptop, etc. At this time, at least one manufacturing device (400) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one manufacturing device (400) may include all kinds of handheld-based wireless communication devices, such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, etc.

[0031] FIG. 2 is a block diagram for explaining a recommendation service providing server included in the system of FIG. 1, and FIG. 3 is a diagram for explaining an embodiment in which an AI-based personalized cosmetic recommendation service according to an embodiment of the present invention is implemented.

[0032] Referring to FIG. 2, the recommendation service providing server (300) may include a receiving unit (310), a de-identification unit (320), an analysis unit (330), a recommendation unit (340), a local utilization unit (350), a distributed storage unit (360), a modeling unit (370), a database unit (380), a custom manufacturing unit (390), and a part-by-part management unit (391).

[0033] When a recommendation service providing server (300) or another server (not shown) operating in conjunction with one embodiment of the present invention transmits an AI-based personalized cosmetic recommendation service application, program, app page, web page, etc. to at least one user terminal (100) and at least one manufacturing device (400), the at least one user terminal (100) and at least one manufacturing device (400) may install or open the AI-based personalized cosmetic recommendation service application, program, app page, web page, etc. Additionally, a service program may be operated on at least one user terminal (100) and at least one manufacturing device (400) using a script executed in a web browser. Here, a web browser refers to a program that enables the use of web (WWW: World Wide Web) services and receives and displays hypertext described in HTML (Hyper Text Mark-up Language), and includes, for example, Chrome, Microsoft Edge, Safari, Firefox, Whale, UC Browser, etc. In addition, "application" refers to an application on a terminal, and includes, for example, an app running on a mobile terminal (smartphone).

[0034] Referring to FIG. 2, the receiving unit (310) can receive a face image from a user terminal (100). At this time, the user terminal (100) can capture the face as an image, and the burden on the server can be reduced by using a method of collecting only data that changes according to changes in light or shooting direction within the image and receiving it as a face image. Detailed information regarding this will be described later in the local utilization unit (350).

[0035] The de-identification unit (320) can store and manage data by dividing the face of the face image into at least one skin zone. The face is divided into various zones according to the skin care area, for example, there are T-zones and U-zones. The T-zone is an area prone to shine and refers to the area extending from the forehead above the eyebrows to the nose. The U-zone refers to the cheeks and chin. The T-zone is an area with a lot of oil, and the U-zone is an area prone to acne. Also, the butterfly zone, which connects the eye area, cheekbones, and cheek area, refers to the area just above the nasolabial folds; it is an area where blackheads need to be managed due to the protruding bridge of the nose, and where pore care is also needed as pores are often enlarged. In this way, the face can be divided according to the areas requiring skin care, and the face images for each skin zone divided in this manner can be stored and managed in a distributed manner, such as a blockchain.

[0036] <Image Splitting>

[0037] In one embodiment of the present invention, a method of dividing only the skin area when dividing the face may be further utilized. For example, the T-zone may not include the eyes, which are best used to identify a person, the U-zone may be extracted centering on the jaw and may not include the lips, and the butterfly zone may not include the eyes or lips. That is, the de-identification unit (320) divides the user's face image into at least one skin zone among the T-zone, U-zone, and butterfly zone, and distributes and stores and manages the face images divided according to the skin zones. In the case of the face image divided into the T-zone, the eyes that are used to identify a person are not included; in the case of the face image divided into the U-zone, the lips are not included; and in the case of the face image divided into the butterfly zone, the eyes and lips are not included, thereby processing to de-identify an individual. The de-identification unit (320) has the advantage of increasing accuracy and speed as the face skin composition data is reduced and the data of the eyes and mouth occupying the face is excluded by dividing and analyzing the face and excluding the eyes and mouth that are used to identify personal information. At this time, although there is a method of segmentation based on color, accurate detection was impossible because the color of the same skin can vary depending on the direction of light, shading, or the angle of the face. Accordingly, in one embodiment of the present invention, a skin region segmentation model can be used as a ViT (Vision Transformer) based model.

[0038] This ViT model basically follows the structure of SINet, but can have a structure where only the encoder part is changed to MobileNet. SINet (Extreme Lightweight Portrait Segmentation Networks with Spatial Squeeze Modules and Information Blocking Decoder) is an ultra-lightweight model for upper body segmentation used as a preprocessing step, similar to skin segmentation. For detailed information on SINet, please refer to the paper (Park, Hyojin, Lars Sjosund, YoungJoon Yoo, Nicolas Monet, Jihwan Bang, and Nojun Kwak "Sinet: Extreme lightweight portrait segmentation networks with spatial squeeze module and information blocking decoder" In Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pp.2066-2074. 2020.).

[0039] At this point, one of the key points of the aforementioned paper is a technique called Information Blocking to reduce the segmentation error of the model. It is stated that Information Blocking eliminates model ambiguity by providing additional information using high-resolution features to areas where the model is not clearly certain, such as boundary regions. Information Blocking is as shown in Equation 1 below.

[0040]

[0041] In Equation 1, Xlow refers to the feature obtained by performing Pointwise Convolution and Bilinear Upsampling on the final feature of the model's encoder to make it the same size as the high-resolution feature. Here, upsampling refers to representing data as if it were collected more frequently than it actually was; in the context of imaging, this means restoring a low-quality image to a high-quality one. Xhigh represents the high-resolution feature, and ⊙ represents the Elementwise Product. The maximum value obtained by applying the softmax to the feature can be viewed as a score indicating how confident the model is in determining whether a corresponding pixel is a skin area. The meaning of subtracting 1, multiplying it by the high-resolution feature, and adding the result is that additional information is not provided to areas where the model is confident, while additional information is provided to areas where the model is uncertain, thereby eliminating model ambiguity and reducing segmentation errors.

[0042] In one embodiment of the present invention, the encoder part of SINet is changed to MobileNet. MobileViT is a model that uses MobileNet-V2 (MV2) blocks and Transformers together, and is a lightweight model friendly to mobile environments. In MobileViT, local information is encoded using a Convolutional Neural Network (CNN), global information is encoded using a Transformer, and then fused and utilized. Furthermore, by maintaining folding or unfolding without compromising the position information of patches, position embedding, which was essential in the existing ViT, is not required.

[0043] Unlike CNNs, ViTs lack inductive bias in images, requiring large datasets and models that need to be deeper and wider. However, MobileViT utilizes CNNs to encode local information and allows for computation through Transformers, enabling it to learn global information while possessing characteristics similar to CNNs. Consequently, it is lighter than other ViT models and allows for model construction with a small number of parameters. Furthermore, to improve performance, the Restormer's GDFD (Gated Depthwise Convolution Feed Forward Network) can be used instead of the simple MobileViT block's Feed Forward Network, and the activation function can be changed to GELU (Gaussian Error Linear Unit), which is the best performing activation function primarily used in Transformers.

[0044] However, since MobileNViT is a lightweight ViT but does not require a small amount of computation, an attention mechanism with less computational computation must be applied. Accordingly, in one embodiment of the present invention, Simplified Channel Attention (SCA) may be used. SCA is an attention mechanism that simplifies channel attention, capable of capturing global information while offering good computational efficiency. The formula is as shown in Equation 2 below.

[0045]

[0046] Here, 'pool' refers to Global Average Pooling and is used to integrate spatial information into channels. * represents the Channelwise Product. X represents a feature, W1 and W2 represent the weights of the CNN, and σ represents the Sigmoid function. If we exclude the Channelwise Product from Equation 2 and substitute the rest with the function f, it can be equal to Equation 3.

[0047]

[0048] Equation 3 is similar to Equation 4, which is the Equation of the GLU (Gate Linear Unit).

[0049]

[0050] Referring to Equation 4, these two equations are similar except for the product of elements. Therefore, let us consider channel attention as a special case of GLU and simplify it into Equation 5.

[0051]

[0052] When applying SCA, the process of folding or unfolding patches for attention operations becomes unnecessary in the model design, allowing the folding or unfolding process to be eliminated. The settings for model training can largely follow the default settings of MobileNet. For the model's loss function, the average of Cross Entropy and DICE coefficients can be used by referring to the experimental results of the Skinny model; additionally, to learn boundaries better, a part calculating the model loss using only boundary components can be added, similar to SINet. The final loss function is as shown in Equation 6.

[0053]

[0054] Here, n is the number of pixels, and yi and hat yi are the label and predicted label of the i-th pixel, respectively. yiedge and hat yiedge are the label and predicted label of the i-th pixel of the boundary component image for the input image, respectively. CE represents the cross-entropy loss function, and DICE represents the DICE loss function using DICE coefficients.

[0055] Accordingly, by processing both local and global information together to suit image segmentation problems, it is possible to achieve high performance while maintaining low computational load. Errors can be reduced by utilizing techniques that provide additional information to regions where the model is uncertain, such as boundary components. Furthermore, by partially adding calculations to the loss function using only boundary components, sharp segmentation of boundaries can be achieved. For this segmentation, SAM (Segment Anything), recently proposed by Meta (formerly Facebook), can be used, or other tools may be utilized.

[0056] The analysis unit (330) can analyze the skin of at least one skin zone using a pre-established analysis model. The analysis model may be a model that analyzes the skin within a face image using a dataset labeled [face image-skin condition] and outputs the skin condition as an analysis result. In this case, the skin condition may be expressed as an index. For example, if there is an image of oily skin, the analysis result may be output as a shine index, if there is an image of large pores, the pore index may be increased and output, or if there is acne or pimples, the trouble index may be increased and output. Also, if there are wrinkles, the wrinkle index may be increased and output, and if there is peeling, the dryness index may be increased and output. The analysis model may be, for example, an OpenCV-based deep learning model, but is not limited thereto.

[0057] The recommendation unit (340) can extract and recommend cosmetics corresponding to the analysis results of the analysis model. The user terminal (100) can take a picture of its face and receive a recommendation for a mask pack. At this time, it may be extracted from a database where [skin condition - mask pack] is mapped one-to-one and stored, or if there is no skin condition in the database, it may find the most similar skin condition and extract a mask pack corresponding to it. Alternatively, it may be a recommendation model that has been trained, verified, and modeled with a dataset of skin condition - mask pack. When using the recommendation model, the mask pack may be recommended by further considering not only the skin condition but also the user's skin concerns, age, gender, preferred ingredients, ingredients to avoid, and allergy ingredients. If the skin concern is elasticity, the proportion of elasticity essence may be increased, and if there is an allergy to some of the whitening ingredients, those ingredients may be excluded.

[0058] The local utilization unit (350) can receive a face image from a user terminal (100), and then remove identical shapes and patterns from the face images that make up the face image captured by the user terminal (100), and receive only face images of different shapes and patterns from the user terminal (100). For example, when a user captures their own face in a video, by uploading only the parts that change depending on the angle or direction of the face capture or shadows to the server, the server does not have to receive a large amount of face images, and by processing repetitive patterns locally, that is, at the user terminal (100), the load on the server can be reduced.

[0059] Skin Analysis Method Using Image Processing

[0060] Image-based skin analysis methods refer to technologies that quantitatively measure skin condition by capturing and analyzing images of the skin surface. Skin condition is assessed by analyzing features extractable from surface images, such as skin tone (color uniformity), pores, texture (roughness) or skin texture, redness (erythema), wrinkles (lines), sebum (oiliness), pigmentation (spots), and elasticity. Skin characteristics are classified in various ways depending on the skin condition.

[0061] Here, skin color can be analyzed by examining one or more of the uniformity of skin tone and the distribution and degree of pigmentation (such as melasma and blemishes). Melanin index can be measured to establish skin brightness or the degree of pigmentation as objective indicators for analysis. Pores include pore count, pore size, and pore distribution, and one or more of these can be analyzed. Pore count measures the number of pores per unit area, pore size measures the width of a single pore to determine the degree of pore enlargement, and pore distribution analyzes how densely pores are concentrated in specific areas of the face. Keratin evaluates the condition of skin texture by analyzing one or more of the unevenness, roughness, distribution, and amount of keratin on the skin surface. Erythema measures the degree of skin redness and identifies erythema caused by vasodilation, inflammation, or sensitivity. Wrinkles evaluate skin elasticity and the degree of aging by analyzing one or more of the depth, length, and number of wrinkles. Sebum assesses skin oiliness by indirectly measuring sebum secretion in specific facial areas, such as the T-zone and U-zone. Pigmentation analyzes one or more of the following: size, number, or concentration of pigmentation, including melasma, blemishes, and acne scars. Skin elasticity evaluates firmness by indirectly measuring the speed at which the skin returns to its original state or the degree of deformation.

[0062] Table 1 below illustrates the classification according to the aforementioned skin conditions as an example, but is not limited thereto and can be used for skin analysis based on various classification conditions.

[0063] Skin Color Skin Tone Uniformity: Uniform / Non-uniform Distribution of Pigmentation: Localized / General Distribution Degree of Pigmentation: Light (Mild) / Dark (Severe) Pores Pore Size (1): Dry / Oily Pore Size (2): Wide / Moderate / Narrow Number of Pores (1): Dry / Oily Number of Pores (2): Many / Moderate / Few Pore Distribution: T-zone concentrated / General distribution / U-zone concentrated Skin Texture (Skin Texture): Smoothness of Skin Texture: Smooth (Uniform) / Rough (Non-uniform / Uneven) Amount of Skin Texture: Low (Refined) / High (Flat) Distribution of Skin Texture: General / Fine distribution / Localized / Concentrated distribution Erythema State of Erythema: Resistance / Sensitivity Erythema Intensity: Normal range / Mild redness / Prominent redness Area of ​​Erythema: None / Localized / Center of T-zone / U-zone / General distribution Wrinkles (Skin Elasticity and Degree of Aging): Of wrinkles Amount (Density) Low Average High Type / Depth of Wrinkles Fine wrinkles Thick wrinkles Deep and long wrinkles Sebum (Oil) Sebum secretion amount by facial area Dry (Dry) Moderate (Normal / Combination) Excessive (Oily) Pigmentation Amount / Number of Pigmentation Low / None Average High Type / Depth of Pigmentation Light / Epidermal Mixed Pigmentation Dark / Dermal Pigmentation Skin Elasticity Skin Elasticity Condition Low Elasticity Good Elasticity

[0064] At this stage, image preprocessing can utilize a light removal process to equalize image brightness by eliminating areas affected by light exposure, a thresholding process to distinguish desired features, and finally, morphological techniques to remove noise—elements other than the desired features. Additionally, since skin features searched in skin surface images, such as pores, have lower brightness compared to surrounding skin, thresholding is applied to enable the differentiation of pores. Furthermore, for image analysis, non-featured surface images can be removed using morphological techniques. To achieve this, an opening method involving erosion followed by dilation can be employed. For feature extraction, a CNN can be used, for example. CNNs demonstrate excellent performance in tasks such as image recognition, object detection, and face recognition, making them suitable for image-based skin condition analysis. CNNs can extract skin features through the process shown in Table 2.

[0065] Convolutional Layer (Role): Detects features in images. It applies filters (kernels) to images to generate feature maps and extracts features such as contours, textures, and colors. Pooling Layer (Role): Reduces spatial size and computational load. It emphasizes features and extracts important information by taking the maximum or average value from sub-regions of the feature map. Fully Connected Layer (Role): Globally learns features to perform final classification. It spreads the CNN output into a single line and performs classification using the fully connected layer, similar to a standard neural network. Dropout (Role): Randomly removes neurons during training to prevent overfitting. It improves the network's generalization performance by randomly removing neurons.

[0066] Of course, it goes without saying that various vision-based analysis algorithms can be used in addition to the CNN mentioned above.

[0067] The distributed storage unit (360) can store face images divided into at least one skin zone in a distributed manner across different storage media. To this end, in one embodiment of the present invention, face images can be stored in a distributed manner using Hadoop-based HDFS, and depending on the embodiment, RAID can also be used. The basic concepts of Hadoop and RAID are explained below.

[0068] <hadoop>

[0069] Hadoop is a large-scale distributed computing framework developed during the development of the open-source web search engine Apache Nutch to rapidly process massive amounts of data at low cost by clustering general-purpose computers and processing data in parallel. Hadoop consists of the Hadoop Distributed File System (HDFS) for storing distributed data and MapReduce for distributed parallel processing of large volumes of data. Its advantage lies in the ability to store and process vast amounts of data at low cost by utilizing inexpensive equipment and storage. Hadoop operates in a Master / Slave structure, consisting of a Master Node that manages slave nodes and Slave Nodes where data is stored. A single Master Node can configure and manage up to 4,096 Slave Nodes. The Master Node manages the namespace and is referred to as the Name Node. Slave Nodes store data in block-unit files and are referred to as Data Nodes.

[0070] HDFS is the underlying storage system used in Hadoop applications. It is a Hadoop distributed file system that supports data streaming access, primarily for processing and storing big data. The NameNode manages metadata such as filenames and replica information. Therefore, when an HDFS client reads data, it receives metadata from the NameNode and retrieves data blocks from the actual DataNodes. When an HDFS client stores data, it receives file control and a list of DataNodes for distributed storage from the NameNode, fragments the data, and stores it on the DataNodes. While storing the fragmented data, the DataNodes store replicas on other DataNodes. In this way, HDFS provides high availability by fragmenting and distributing data across multiple DataNodes.

[0071] <raid>

[0072] RAID (Redundant Array of Inexpensive / Independent Disks) is a technology developed to combine multiple storage devices to achieve the same effect as a single high-capacity, high-performance storage device. It was designed to ensure data availability, security, and ease of expanding disk capacity. RAID is classified into levels 0 through 6 depending on the configuration method. RAID 0, also known as striping, is a distributed storage concept that divides data across multiple storage locations. Therefore, it requires at least two disks for distributed storage and improves speed by enabling parallel data processing. However, since RAID 0 does not provide data redundancy, it has the disadvantage that the RAID system collapses if even a single disk fails. RAID 1, known as mirroring, is a data replication concept that duplicates identical data onto different disks. Therefore, while it requires at least two disks, it allows for replacement with another disk even if one fails.

[0073] RAID 2 through 6 are based on the concept of storing parity information for recovery, and the RAID level is determined by factors such as the striping unit, error checking, and the parity storage method. The striping unit that determines the RAID level is bit, byte, or block, and the RAID level is determined by the method of storing parity information, whether on a separate disk or distributed across multiple disks. Due to the storage of parity information, RAID 2 through 5 consists of at least three disks, while RAID 6 consists of at least four disks. RAID levels can be used in combination; to apply data distributed storage and replication functions similar to Hadoop, for example, data can be distributed across two drives, 1 and 2, using RAID 0, and a replica of the data from drives 1 and 2 can be configured on drives 3 and 4 using RAID 1. While it may be similar to Hadoop in that it can provide high performance and high availability of data through such distributed storage and data replication, RAID is a technology for high performance and high availability of disk-level storage devices rather than file systems. Accordingly, in one embodiment of the present invention, Hadoop is configured as the standard, but RAID may be configured to be used as an auxiliary method depending on the embodiment.

[0074] Returning to the description of the distributed storage unit (360) based on the basic concept described above, in one embodiment of the present invention, Hadoop’s HDFS can be used to distribute face images (divided images) divided into skin zones to different storage media. In existing environments, if an attacker succeeds in a data leakage attack, they can obtain the data as is, and even if the data is encrypted, if they succeed in decrypting it, the original data can be leaked as is. On the other hand, as described above, if the face image is divided and fragmented (first de-identification) and the fragmented face image is distributed again (second de-identification) to achieve double de-identification through [fragmentation-distribution], the attacker cannot obtain the original data because they only obtain a portion of the fragmented data even if they succeed in data leakage. Furthermore, by replicating the fragmented data and distributing it to other storage locations, the user can use the original data by combining the fragmented data stored in other storage locations even if some data storage locations are damaged.

[0075] HDFS is a technology designed to store large files in blocks of a specific size, and additional features are required to protect data. For example, HDFS has a default fragmentation size of 128MB; since data smaller than this size is not fragmented, the original, unfragmented data can be leaked. Therefore, ① through the Policy Management function, fragmentation policies can be configured to adjust the fragmentation size so that all data is fragmented, or to fragment data into a fixed number of blocks rather than a specific size. Additionally, ② for data protection, a storage method suitable for the data operating environment must be considered. Because Data Nodes installed on the same physical server as the NameNode offer faster speeds in terms of data I / O, HDFS provides a policy that prioritizes data storage on Data Nodes installed alongside the NameNode. In such cases, if data from that Data Node is leaked, all fragmented data constituting the original data can be compromised. This is because even if the original data is fragmented into A, B, and C, A, B, and C are all stored on the Data Node installed alongside the NameNode.

[0076] Furthermore, ③ the operational environment of the data must also be considered. HDFS uses a policy that stores replicas based solely on the rack information where the data nodes are installed. While this storage policy is designed for operational environments where data is stored across numerous racks, such as data centers, operational data environments do not necessarily configure data storage in racks; they may also be located in isolated locations. Therefore, for data fragmentation and decentralization technologies to be applied for data protection, a distributed storage policy tailored to the actual operational environment of the data must be implemented. To this end, policies such as ① a Fragment Policy that can fragment all data, ② a Replication Policy that determines the number of replicas based on the number of data storage configurations and data importance, and ③ an Encryption Policy that encrypts fragmented data using the ARIA encryption algorithm before fragmenting it to prevent leakage, even if the data is fragmented, can be applied.

[0077] In case ②, for distributed storage, distributed policy management based on isolated locations or positions can be performed instead of Hadoop's rack-based storage method. Additionally, data can be protected by enabling data retrieval and storage by region through authentication and authorization functions. When data is fragmented and distributed in this manner, an internal leaker must physically access multiple data servers to collect fragmented data that can be reassembled into the original. Even when attempting a data leakage attack via a network, a large number of data servers must be controlled to secure fragmented data capable of being assembled into the original. Furthermore, even if all fragmented data is collected, the original data can only be restored by knowing the fragmentation information, and restoring encrypted fragmented data is nearly impossible.

[0078] The modeling unit (370) can model a dataset of face images and skin conditions by learning, verifying, and testing it with at least one artificial intelligence algorithm, and then set the artificial intelligence algorithm with the highest accuracy among the at least one artificial intelligence algorithm as the analysis model. At this time, race, gender, age, etc. can be distinguished using OpenCV, and results that take personal data into account can be obtained using the recommendation system described above.

[0079] The database unit (380) can store mask packs mapped according to skin condition. For example, the skin condition can be variableized (quantified) into a moisture index, a skin trouble index, a whitening index, a wrinkle index, etc., and the ingredients and ratios corresponding to these variables can be stored one-to-one, after which a mask pack having these ingredients and ratios can be extracted. By creating an index corresponding to the variable matching in a fast memory cache and comparing it with a predefined mask pack simultaneously with the matching, a mask pack can be extracted and recommended. Furthermore, when updating information corresponding to the unique ID of the mask pack, it can be applied quickly without affecting existing data.

[0080] The custom manufacturing unit (390) can manufacture a mask pack by mixing and blending materials according to at least one ingredient and ratio of the mask pack in the manufacturing unit (400) when the manufacturing unit (400) for manufacturing a mask pack is linked with the user terminal (100). In addition to recommending a mask pack, it can also manufacture a mask pack using the ingredients and ratios of the aforementioned database.

[0081] The area-specific management unit (391) can extract and recommend cosmetics according to the skin condition of at least one skin zone when the user terminal (100) selects a recommendation by skin zone when recommending cosmetics. For example, if the eye area is dry and wrinkled, the nose area (butterfly zone) is oily, has many blackheads and wide pores, and the U zone, that is, the chin area, has acne or pimples, then a pack containing elasticity ingredients can be recommended for the eye area, a pack containing ingredients that reduce pores and remove blackheads or oiliness can be recommended for the nose area, and a pack containing ingredients that soothe acne or pimples can be recommended for the chin area. At this time, since a skin zone is defined as an area that includes at least one facial area, the cosmetics by skin zone may be packs for each facial area, or packs for an area that combines facial areas. Also, the types of cosmetics are not limited to packs only.

[0082] Hereinafter, the operation process according to the configuration of the recommendation service providing server of FIG. 2 described above will be explained in detail with reference to FIG. 3. However, it is obvious that the embodiment is merely one of the various embodiments of the present invention and is not limited thereto.

[0083] Referring to FIG. 3, (a) when a face is captured by a user terminal (100), the recommendation service providing server (300) can divide the face image into at least one skin zone as shown in (b), analyze the skin as shown in (c), and recommend a mask pack as shown in (d). At this time, recommendations can be made for specific areas of the skin zone, or recommendations can be made for the entire face that covers the whole face. A platform according to one embodiment of the present invention can be built to provide a customized mask pack after measuring the skin, and furthermore, when linked with a manufacturing device (400), it can create a pack suitable for the user, that is, a pack containing ingredients necessary for the user, even if there is no ready-made pack available.

[0084] As for the details regarding the AI-based personalized cosmetic recommendation service provision method of FIGS. 2 and FIGS. 3 that are not described, they are identical to or can be easily inferred from the details regarding the AI-based personalized cosmetic recommendation service provision method described above through FIGS. 1, so further explanation will be omitted.

[0085] FIG. 4 is a diagram illustrating the process of data transmission and reception between each component included in the AI-based personalized cosmetic recommendation service provision system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of the process of data transmission and reception between each component will be described through FIG. 4, but the present invention is not to be interpreted as being limited to such an embodiment, and it is obvious to those skilled in the art that the process of data transmission and reception shown in FIG. 4 may be changed according to various embodiments described above.

[0086] Referring to FIG. 4, the recommendation service provider server receives a face image from a user terminal (S4100).

[0087] And, the recommendation service provider server divides the face of the face image into at least one skin zone and stores and manages the data (S4200), and analyzes the skin of at least one skin zone using a pre-established analysis model (S4300).

[0088] In addition, the recommendation service provider server extracts and recommends cosmetics corresponding to the analysis results of the analysis model (S4400).

[0089] The order of the steps described above (S4100–S4400) is merely an example and is not limited thereto. That is, the order of the steps described above (S4100–S4400) may vary, and some of these steps may be executed simultaneously or deleted.

[0090] As for the details regarding the AI-based personalized cosmetic recommendation service provision method of Fig. 4 that are not explained, they are identical to or can be easily inferred from the details regarding the AI-based personalized cosmetic recommendation service provision method described above through Figs. 1 to 4, so further explanation will be omitted.

[0091] The method for providing an AI-based personalized cosmetic recommendation service according to one embodiment described through FIG. 4 may also be implemented in the form of a recording medium containing computer-executable instructions, such as an application or program module executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and non-removable media. Additionally, a computer-readable medium may include all computer storage media. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0092] The method for providing an AI-based personalized cosmetic recommendation service according to one embodiment of the present invention described above may be executed by an application basically installed on a terminal (which may include a program included in a platform or operating system, etc., basically installed on the terminal), or by an application (i.e., a program) directly installed by a user on a master terminal through an application providing server, such as an application store server, an application, or a web server related to the service. In this sense, the method for providing an AI-based personalized cosmetic recommendation service according to one embodiment of the present invention described above may be implemented as an application (i.e., a program) that is basically installed on a terminal or directly installed by a user, and may be recorded on a computer-readable recording medium such as a terminal.

[0093] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0094] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.< / raid> < / hadoop>

Claims

1. A user terminal that photographs a face and receives cosmetic recommendations; and A recommendation service providing server comprising: a receiving unit for receiving a face image from the user terminal; a de-identification unit for dividing the face of the face image into at least one skin zone and storing and managing data; an analysis unit for analyzing the skin of the at least one skin zone using a pre-established analysis model; and a recommendation unit for extracting and recommending cosmetics corresponding to the analysis results of the analysis model; An AI-based personalized cosmetics recommendation service provision system including 2. In Paragraph 1, The above recommendation service providing server is, A local utilization unit that, when receiving a face image from the user terminal, removes identical shapes and patterns from among the face images constituting the face image captured by the user terminal, and then receives only face images of different shapes and patterns from the user terminal; An AI-based personalized cosmetic recommendation service provision system characterized by further including 3. In Paragraph 1, The above recommendation service providing server is, A distributed storage unit for storing face images divided into at least one skin zone in a distributed manner on different storage media; An AI-based personalized cosmetic recommendation service provision system characterized by further including 4. In Paragraph 1, The above recommendation service providing server is, A modeling unit that models a dataset of face images and skin conditions by learning, verifying, and testing it with at least one artificial intelligence algorithm, and then sets the artificial intelligence algorithm having the highest accuracy among the at least one artificial intelligence algorithm as the analysis model; An AI-based personalized cosmetic recommendation service provision system characterized by further including 5. In Paragraph 1, The above cosmetic product includes a mask pack, The above recommendation service providing server is, A database unit that stores mask packs mapped according to skin condition; An AI-based personalized cosmetic recommendation service provision system characterized by further including 6. In Paragraph 1, The above cosmetic product includes a mask pack, The above recommendation service providing server is, When the manufacturing device for manufacturing the above mask pack is linked with the above user terminal, a customized manufacturing unit that manufactures the mask pack by mixing and blending materials according to at least one component and ratio of the above mask pack in the manufacturing device; An AI-based personalized cosmetic recommendation service provision system characterized by further including 7. In Paragraph 1, The above recommendation service providing server is, When recommending the above cosmetics, if a skin zone recommendation is selected on the user terminal, a body-specific management unit that extracts and recommends cosmetics according to the skin condition of at least one skin zone; An AI-based personalized cosmetic recommendation service provision system characterized by further including 8. In Paragraph 1, The above-mentioned de-identification unit divides a user's face image into at least one skin zone among a T-zone, a U-zone, and a butterfly zone, and stores and manages the face images divided according to the skin zones in a distributed manner. In the case of a face image divided by the T-zone, it does not include eyes that can identify a person, and In the case of a face image divided into U-zones, the lips are not included, In the case of a face image divided by the butterfly zone, exclude the eyes and lips, AI-based personalized cosmetic recommendation service provision system characterized by the ability to process so as to de-identify individuals.

Images