Image processing method and apparatus, and electronic device and storage medium

Abstract

Provided in the present disclosure are an image processing method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring relative positional relationship information between an intraoral model and a neutral facial model of a target user; using base data of basic expressions to form a plurality of pieces of facial data corresponding to expression states; using the facial data to update the expression state of the neutral facial model; and on the basis of the relative positional relationship information, superimposing a portion, for which an aesthetic design has not been generated, in the intraoral model and a pre-constructed dental model onto the neutral facial model, the expression state of which has been updated, so as to obtain an expression image showing an aesthetic design effect. In the present solution, aesthetic restoration effects under different expressions are dynamically displayed to a target user by means of expression images showing aesthetic design effects, thereby improving the display effect of the aesthetic restoration effects, and the user experience.

Description

An image processing method, apparatus, electronic device, and storage medium

[0001] This disclosure claims priority to Chinese Patent Application No. 202411671660.2, filed on November 20, 2024, entitled “An Image Processing Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of image processing technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0003] With people's pursuit of health and aesthetics, many patients with poor tooth shape hope to improve their tooth shape through aesthetic restoration. Before formally performing aesthetic restoration, it is necessary to show the patient the effect of the restoration in advance, and only after the patient is satisfied will the formal aesthetic restoration be performed on the patient's teeth.

[0004] Currently, the method for demonstrating the effects of aesthetic restorations to patients involves using aesthetic design software to create a dental model, which is then overlaid onto the patient's facial image to simulate how the patient will look after the restoration. However, this method only displays the designed dental model statically, resulting in patients not being able to fully understand the effects of the aesthetic restoration, leading to poor presentation of the results and a poor patient experience. Summary of the Invention

[0005] In view of this, the present disclosure provides an image processing method, apparatus, electronic device, and storage medium to solve the problems of poor display of aesthetic restoration effects and poor patient experience in current methods of displaying aesthetic restoration effects to patients.

[0006] To achieve the above objectives, the present disclosure provides the following technical solutions:

[0007] The first aspect of this disclosure discloses an image processing method, the method comprising:

[0008] The relative positional relationship information between the intraoral model and the neutral facial model of the target user is obtained. The neutral facial model is a three-dimensional facial model obtained by scanning the target user in a non-expression state.

[0009] Using the base data of basic facial expressions, multiple facial data corresponding to the facial expression states are formed. The base data of basic facial expressions is constructed based on a neutral facial model.

[0010] The facial expression state of the neutral facial model is updated multiple times using facial data;

[0011] Based on relative positional information, the parts of the intraoral model without aesthetic design and the pre-built tooth model are superimposed on a neutral facial model with updated facial expression state to obtain an expression image that demonstrates the effect of the aesthetic design.

[0012] Preferably, the intraoral model includes a maxillary model, a mandibular model, and occlusal relationship data;

[0013] Based on relative positional information, the parts of the intraoral model without aesthetic design and the pre-built dental model are superimposed onto a neutral facial model that has been updated with facial expressions to obtain an expression image that showcases the effect of the aesthetic design, including:

[0014] Identify the feature point information of a neutral facial model, calculate the positional changes of specific feature points, and obtain the transformation matrix.

[0015] Based on the relative positional relationship information and the transformation matrix, the updated position of the mandibular model in the neutral facial model with updated expression state is obtained;

[0016] Based on relative positional information and update position, the parts of the intraoral model that have not generated aesthetic design and the pre-built tooth model are superimposed on the neutral facial model that has been updated with facial expression state to obtain an expression image that shows the effect of aesthetic design.

[0017] Preferably, using the base data of basic expressions, multiple facial data corresponding to the expression state are formed, including:

[0018] Capture the target user's facial expressions to obtain multiple frames of images to be processed;

[0019] Using the base data of multiple frames of images to be processed and basic facial expressions, multiple facial data corresponding to the multiple frames of images to be processed are generated.

[0020] Preferably, multiple facial data corresponding to the multiple frames of images to be processed are formed using base data of multiple frames of images to be processed and basic facial expressions, including:

[0021] Based on multiple frames of images to be processed, determine the basic facial expressions associated with the multiple frames of images to be processed, and determine the weight parameters of the basic facial expressions associated with the multiple frames of images to be processed. The multiple frames of images to be processed contain RGB data and depth data.

[0022] By using the base data and weight parameters of the basic expressions associated with multiple frames of images to be processed, multiple facial data corresponding to the multiple frames of images to be processed are formed.

[0023] Preferably, the part of the intraoral model for which no aesthetic design has been generated is the intraoral model outside the maxillary anterior tooth region, and the pre-constructed tooth model is the tooth model of the maxillary anterior tooth region.

[0024] Preferably, before overlaying the portion of the intraoral model from which no aesthetic design has been generated and the pre-built dental model onto the neutral facial model whose expression state has been updated, the method further includes:

[0025] Aesthetic design software is used to generate dental models designed for the target users as pre-built dental models.

[0026] Preferably, using the base data of basic expressions, multiple facial data corresponding to the expression state are formed, including:

[0027] By using weight parameters of basic facial expressions related to the facial expression change images of other users, multiple facial data corresponding to the facial expression state are formed. The facial expression change images of other users are obtained by capturing the facial expression change of other users.

[0028] Preferred options also include:

[0029] Generate 3D animations, which include: animations with aesthetic design effects consisting of multiple frames of facial expression images, and / or animations without aesthetic design effects consisting of multiple frames of images to be processed.

[0030] Preferably, the relative positional relationship information between the intraoral model and the neutral facial model of the target user is obtained, including:

[0031] Acquire the smiling face model, intraoral model and neutral face model of the target user. The smiling face model is a 3D face model obtained by scanning the target user in a toothy smiling expression state.

[0032] The intraoral model is aligned to the smiling face model, and the neutral face model is aligned to the smiling face model to obtain the relative positional relationship information between the intraoral model and the neutral face model.

[0033] Preferably, the process of constructing base data for basic facial expressions based on a neutral facial model includes:

[0034] Process the mesh structure at the lips of the neutral face model to separate the upper and lower lips of the neutral face model;

[0035] Identify and process feature point information of the neutral facial model;

[0036] Based on the processed neutral facial model and feature point information, base data for multiple basic expressions are constructed through deformation transfer.

[0037] A second aspect of this disclosure discloses an image processing apparatus, the apparatus comprising:

[0038] The acquisition unit is configured to acquire the relative positional relationship information between the intraoral model and the neutral facial model of the target user. The neutral facial model is a three-dimensional facial model obtained by scanning the target user in a non-expression state.

[0039] The forming unit utilizes the base data of basic expressions to generate multiple facial data corresponding to the expression state. The base data of basic expressions is constructed based on a neutral facial model.

[0040] The update unit is configured to update the expression state of the neutral facial model multiple times using facial data;

[0041] The overlay unit is configured to overlay the portion of the intraoral model without aesthetic design and the pre-built tooth model onto a neutral facial model that has been updated with facial expression state, based on relative positional relationship information, to obtain an expression image that demonstrates the effect of the aesthetic design.

[0042] A third aspect of this disclosure discloses an electronic device, including a processor and a memory, which are connected via a communication bus; wherein the processor is configured to call and execute a program stored in the memory; and the memory is configured to store a program that implements the image processing method disclosed in the first aspect of this disclosure.

[0043] The fourth aspect of this disclosure discloses a computer-readable storage medium storing computer-executable instructions configured to perform the image processing method disclosed in the first aspect of this disclosure.

[0044] The fifth aspect of this disclosure discloses a computer program product, including a computer program that, when executed by a processor, implements the methods of any of the above aspects.

[0045] Based on the above-described embodiments of this disclosure, an image processing method, apparatus, electronic device, and storage medium provide the following method: acquiring relative positional relationship information between an intraoral model and a neutral facial model of a target user; using basic expression data to form multiple facial data corresponding to the expression state; updating the expression state of the neutral facial model using the facial data; and, based on the relative positional relationship information, superimposing the portion of the intraoral model from which no aesthetic design has been generated and a pre-constructed dental model onto the updated neutral facial model to obtain an expression image showcasing the aesthetic design effect. In this solution, the expression state of the neutral facial model is updated. The portion of the intraoral model from which no aesthetic design has been generated and a pre-constructed dental model are superimposed onto the updated neutral facial model to obtain an expression image showcasing the aesthetic design effect, thereby improving the display effect of aesthetic restoration and the user experience. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 is a flowchart of an image processing method provided in an embodiment of this disclosure;

[0048] Figure 2 is an example diagram of an intraoral model provided in an embodiment of this disclosure;

[0049] Figure 3 is an example diagram of a smiling face model provided in an embodiment of this disclosure;

[0050] Figure 4 is an example diagram of a neutral facial model provided in an embodiment of this disclosure;

[0051] Figure 5 is another example diagram of a smiling face model provided in an embodiment of this disclosure;

[0052] Figure 6 is an example diagram of separating the upper and lower lips of a neutral facial model according to an embodiment of this disclosure;

[0053] Figure 7 is an example image of the lips of a neutral facial model before processing, provided in an embodiment of this disclosure.

[0054] Figure 8 is an example diagram of the lips of a neutral facial model after processing, provided in an embodiment of this disclosure;

[0055] Figure 9 is an example diagram of facial feature points of the processed neutral face model provided in the embodiments of this disclosure;

[0056] Figure 10 is an example diagram of the basic facial expressions provided in the embodiments of this disclosure;

[0057] Figure 11 is another example diagram of the basic facial expressions provided in the embodiments of this disclosure;

[0058] Figure 12 is an example diagram of the DSD aesthetic design model provided in the embodiments of this disclosure;

[0059] Figure 13 is an example diagram of facial expression images in a three-dimensional animation provided in an embodiment of this disclosure;

[0060] Figure 14 is a structural block diagram of an image processing apparatus provided in an embodiment of this disclosure. Detailed Implementation

[0061] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0062] In this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0063] As the background technology indicates, with people's pursuit of health and aesthetics, many patients with poor dental morphology hope to improve their teeth through aesthetic restorations, such as Digital Smile Design (DSD), to enhance their dental shape, showcase a brighter smile, and thus improve their self-confidence. Before formally performing aesthetic restorations, it is necessary to demonstrate the desired effect to the patient in advance, and only after the patient is satisfied will the formal aesthetic restoration proceed.

[0064] Currently, the method for demonstrating the effects of aesthetic restorations to patients involves designing a dental model using aesthetic design software (such as DSD aesthetic design software), and then overlaying the dental model onto the patient's facial image to simulate the patient's appearance after the aesthetic restoration. In other words, at present, it simply involves overlaying the patient's dental design data (or dental simulation design data) onto the patient's two-dimensional facial image to simulate the patient's appearance after the aesthetic restoration.

[0065] However, the existing methods for showcasing the effects of aesthetic restorations can only statically display the designed dental model. This results in patients not being able to fully understand the effects of aesthetic restorations, leading to poor presentation of the effects and a poor patient experience.

[0066] To address the aforementioned issues, this solution proposes an image processing method, apparatus, electronic device, and storage medium to update the expression state of a neutral facial model. The portion of the intraoral model without aesthetic design and a pre-constructed dental model are superimposed onto the updated neutral facial model, thereby obtaining an expression image showcasing the aesthetic design effect, improving the display effect of aesthetic restoration and enhancing the user experience.

[0067] In other words, this approach incorporates a series of facial expressions and movements of patients requiring aesthetic restoration into the presentation of the restoration results, showcasing the aesthetic restoration effects of the patients under these expressions and movements in advance, thus more vividly demonstrating the final appearance of the patients after the aesthetic restoration is completed.

[0068] The following describes the solution in detail through various embodiments.

[0069] It should be noted that the subsequent explanation of this plan will involve scanning the target user's 3D facial model and other personal information. Before scanning the target user's 3D facial model, this plan has informed the target user and obtained the target user's authorization for the collection of personal information. That is, this plan collects the target user's 3D facial model and other personal information in compliance with relevant laws and regulations.

[0070] Referring to Figure 1, a flowchart of an image processing method provided by an embodiment of this disclosure is shown, the method including:

[0071] Step S101: Obtain the relative positional relationship information between the intraoral model and the neutral facial model of the target user.

[0072] It should be noted that the target users are patients who need dental aesthetic restoration.

[0073] In the specific implementation step S101, the target user's smiling face model, intraoral model, and neutral face model are obtained. The smiling face model is a three-dimensional face model obtained by scanning the target user in a toothy smiling expression state. The neutral face model is a three-dimensional face model obtained by scanning the target user in a non-expression state. The intraoral model includes the maxillary model, mandibular model, and occlusal relationship data.

[0074] The method for obtaining the smiling facial model, intraoral model, and neutral facial model of the target user is as follows: the target user's intraoral model is obtained by scanning the target user's intraoral space using a scanning device; specifically, the target user's maxilla and mandible are scanned using a scanning device to obtain the target user's maxillary and mandibular models, as well as the occlusal relationship data.

[0075] It should be noted that during the scanning of the target user's occlusal data, the target user's maxilla and mandible need to be aligned in the occlusal relationship. The occlusal data obtained from the scan can be used as a reference benchmark for "changes in the occlusal position of the maxilla and mandible" during the subsequent facial expression change processing.

[0076] For example, the scan yields the intraoral model of the target user as shown in Figure 2, which includes the maxillary model, mandibular model, and occlusal relationship data.

[0077] The target user is in a smiling expression with teeth showing. The 3D facial model of the target user is scanned by a scanning device to obtain the smiling facial model of the target user.

[0078] It should be noted that during the scanning of the target user's smiling facial model, the target user's teeth must maintain a normal occlusal relationship (the maxilla and mandible must be aligned), that is, the occlusal relationship corresponds to the occlusal relationship when the intraoral model is obtained.

[0079] For example, when the target user's teeth maintain a normal occlusal relationship and the target user is in a smiling expression with teeth showing, the scanning yields the smiling facial model of the target user as shown in Figure 3.

[0080] The target user is in a neutral facial state (or with their mouth naturally closed). The three-dimensional facial model of the target user is scanned by a scanning device to obtain a neutral facial model of the target user.

[0081] For example, when the target user is in a normal, closed-mouth state, a neutral facial model of the target user is obtained by scanning, as shown in Figure 4.

[0082] The above methods can be used to scan and obtain the target user's smiling facial model, intraoral model, and neutral facial model.

[0083] It should be noted that data collected by different scanning devices are usually not aligned, and data collected multiple times by the same scanning device are also usually not aligned. Therefore, after obtaining the smiling face model, intraoral model, and neutral face model of the target user, it is necessary to align the smiling face model, intraoral model, and neutral face model of the target user.

[0084] In the specific implementation, the intraoral model of the target user is aligned to the smiling face model of the target user, and the neutral face model of the target user is aligned to the smiling face model of the target user, so as to obtain the relative positional relationship information between the intraoral model and the neutral face model.

[0085] In other words, the intraoral model of the target user is first aligned to the smiling face model, and then the neutral face model is aligned to the smiling face model. This indirectly aligns the intraoral model to the neutral face model, thereby obtaining the relative positional relationship information between the intraoral model and the neutral face model.

[0086] It should be noted that the alignment operations of "aligning the intraoral model to the smiling face model" and "aligning the neutral face model to the smiling face model" can be implemented using a mesh matching algorithm. That is, the mesh matching algorithm is used to align the intraoral model to the smiling face model and the neutral face model to the smiling face model.

[0087] In some embodiments, the grid matching algorithm includes coarse grid matching algorithm, fine grid matching algorithm, etc., such as PCA, ICP and other grid matching algorithms.

[0088] To better understand the alignment operation mentioned above, the following example illustrates the process: Using the tooth information from the target user's smiling face model (toothy smile) as shown in Figure 5, the target user's maxillary and mandibular models are aligned onto the smiling face model. Then, the target user's neutral face model is aligned onto the smiling face model. Thus, through the target user's smiling face model, the neutral face model, maxillary model, and mandibular model are indirectly aligned, thereby obtaining the relative positional relationship information between the intraoral model and the neutral face model. This relative positional relationship information reflects the correct positional relationship between the tooth model and the neutral face model.

[0089] Step S102: Using the base data of basic expressions, generate multiple facial data corresponding to the expression state.

[0090] It should be noted that the base data for basic facial expressions is constructed based on a neutral facial model; in some embodiments, the specific method for constructing the base data for basic facial expressions based on the neutral facial model of the target user is as follows:

[0091] The mesh structure at the lips of the neutral face model is processed to separate the upper and lower lips of the neutral face model; the feature point information of the processed neutral face model (the neutral face model after separating the upper and lower lips) is identified; based on the processed neutral face model and the feature point information, the base data of multiple basic expressions is constructed through deformation transfer.

[0092] Specifically, a mesh geometry processing algorithm is used to process the mesh structure at the lips of the neutral face model, thereby separating the upper and lower lips of the neutral face model.

[0093] For example, by using a mesh geometry processing algorithm to process the mesh structure at the lips of a neutral face model, an example image showing the separation of the upper and lower lips of the neutral face model, as shown in Figure 6, can be obtained; the state of the lips of the neutral face model before processing is shown in Figure 7, and the state of the lips of the neutral face model after processing is shown in Figure 8.

[0094] After separating the upper and lower lips of the neutral face model, the facial feature points of the processed neutral face model are identified by AI recognition (or by specified software) to obtain the corresponding feature point information.

[0095] For example, facial feature points of a neutral facial model, as shown in Figure 9, can be identified through AI recognition, thereby obtaining the corresponding feature point information.

[0096] Using the processed neutral facial model and the feature point information of the recognized neutral facial model, the base data of the target user's basic expressions is constructed through deformation transfer. The base expressions can also be called expression base.

[0097] The base data for multiple basic expressions is constructed through deformation transfer of expression base templates. The expression base template library contains at least one single expression such as left eye blinking, right eye blinking, open mouth, and pouting. Expression base templates are selected from the expression base template library, and deformation transfer is used to construct the base data for multiple basic expressions corresponding to the target user. For example, Figure 10 shows an example image of the basic expression "open mouth," and Figure 11 shows an example image of the basic expression "pouting." When using these basic expressions, the corresponding basic expression is selected from these basic expressions.

[0098] The base data of the target user's basic expressions are numbered, for example, A1, A2 and A3 to represent the base data of different basic expressions; the base data of each basic expression has the same mesh topology as the neutral face model.

[0099] In the specific implementation of step S102, the base data of the pre-constructed basic expressions are used to form multiple facial data corresponding to the expression state. In the specific implementation, this solution provides at least two ways to form facial data, which are illustrated below.

[0100] The first method for generating facial data involves capturing images of the target user's facial expressions to obtain multiple frames of images to be processed. Using these multiple frames of images and the base data of the basic facial expressions, multiple facial data corresponding to the multiple frames of images to be processed are generated. The facial data corresponding to the images to be processed consists of the base data of the basic facial expressions and weight parameters.

[0101] Specifically, the process involves capturing images of the target user's facial expressions to obtain multiple frames of images (facial expression data) that can be used for facial expression tracking calculations. These images contain RGB data and depth data, where RGB refers to red, green, and blue.

[0102] When the target user's facial expression changes, each frame of the image to be processed is acquired for facial expression tracking calculation.

[0103] In the specific implementation, the method of forming multiple facial data using multiple frames of images to be processed and the base data of basic expressions is as follows: Based on multiple frames of images to be processed, determine the basic expressions related to the multiple frames of images to be processed, and determine the weight parameters of the basic expressions related to the multiple frames of images to be processed. The multiple frames of images to be processed contain RGB data and depth data; using the base data and weight parameters of the basic expressions related to the multiple frames of images to be processed, multiple facial data corresponding to the multiple frames of images to be processed are formed.

[0104] It should be noted that multiple frames of images to be processed can be captured, and each frame of the image to be processed can form a corresponding facial data. That is, one frame of the image to be processed can yield one facial data.

[0105] It should be further noted that the image to be processed contains RGB data and depth data. For each frame of the image to be processed, the expression state of a three-dimensional facial model can be obtained through the RGB data and depth data in that frame. The expression state of each frame of the three-dimensional facial model can be expressed by the basic data and weight parameters of the basic expression associated with that frame of the image to be processed. The expression state expressed by the basic data and weight parameters of the basic expression associated with that frame of the image to be processed is the facial data mentioned above (denoted as A).

[0106] In summary, for each frame of the image to be processed (each frame of RGB data and Depth data), facial data A corresponding to that frame of the image to be processed is formed. Facial data A can be constructed by multiplying the base data of a finite number of base expressions in the pre-constructed base expressions by certain weight parameters. For example, A = c1*A1 + c5*A5 + c8*A8, where A1, A5, and A8 are the base data of the base expressions related to that frame of the image to be processed, and c1, c5, and c8 are the weight parameters corresponding to A1, A5, and A8. The weight parameters ci corresponding to the base data of each base expression are obtained, and the sum of ci is 1.

[0107] In summary, the coordinates of each point in the facial data A corresponding to each frame of the image to be processed are obtained by multiplying the coordinate values ​​corresponding to Ai by ci and summing them. Ai is the base data of the base expression i, and ci is the weight parameter corresponding to the base data of the base expression i.

[0108] The facial data A generated in the above manner has the same mesh topology as the neutral facial model.

[0109] The above is a description of the first method for generating facial data.

[0110] The second method for generating facial data is to generate multiple facial data using images of other users' facial expressions, without needing to capture the target user's facial expression changes (i.e., without capturing the image to be processed or collecting the target user's dynamic facial expressions).

[0111] In practice, by utilizing the base data and weight parameters of basic expressions related to the facial expression change images of other users, multiple facial data corresponding to the expression state are formed (used as the facial data of the target user). That is, it is possible to form multiple facial data of the target user without capturing the facial expression change of the target user, using the base data and weight parameters of basic expressions related to the facial expression change images of other users. It should be noted that using the base data of basic expressions related to the facial expression change images of other users means using the base data of basic expressions related to the facial expression change images of other users to determine the expression base template corresponding to the weight parameters. Through the corresponding expression base template, the expression base template corresponding to each weight parameter when the target user uses the weight parameters of other users, as well as the base data of the target user's basic expressions, can be determined to form the target user's facial data.

[0112] The base data of the basic expressions related to the expression change images of other users were constructed based on a neutral face model, and the expression change images of other users were obtained by capturing the expression change of other users.

[0113] In a practical application, after constructing the base data of each basic expression through the neutral facial model of the target user, the facial data of the target user can be formed by using preset weight parameters, or by using weight parameters obtained from the dynamic expressions of other users. The facial data of the target user can be formed without collecting the dynamic expressions of the target user.

[0114] For example: Suppose the target user is user B; obtain the weight parameter c obtained from user A's dynamic facial expressions, and then use the weight parameter c obtained from user A's dynamic facial expressions to form user B's facial data. User B does not need to collect dynamic facial expressions.

[0115] In another practical application, the combination of base data for basic expressions can be set by changing the weight parameters, thereby forming multiple facial data corresponding to the expression state (equivalent to setting facial data using weight parameters and base data). This allows for dynamic simulation of the aesthetic restoration effect of the target user under different facial expression changes without collecting the target user's dynamic expressions.

[0116] It should be noted that when using other users' facial expression images to form the target user's facial data, you can use only weight parameters, or you can use both base data and weight parameters. However, when forming the target user's facial data, the base data used is generated by the target user's neutral facial model. The base data of other users is only used as a reference expression for selecting the target user's base data corresponding to the weight parameters. This makes the formed target user's facial data more vivid and natural.

[0117] Step S103: Update the expression state of the neutral facial model multiple times using facial data.

[0118] In the specific implementation step S103, after forming the facial data corresponding to each frame of the image to be processed, the facial data corresponding to each frame of the image to be processed is used to update the expression state of the neutral facial model multiple times.

[0119] Step S104: Based on the relative positional relationship information, the part of the intraoral model without aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image that shows the effect of aesthetic design.

[0120] In some embodiments, a pre-built model of the target user's teeth after aesthetic restoration (referred to as a pre-built tooth model) is constructed. Specifically, an aesthetic design software is used to generate a tooth model designed for the target user as a pre-built tooth model.

[0121] For example, using DSD aesthetic design software, a DSD aesthetic design model of the maxillary anterior teeth region of the target user is generated, as shown in Figure 12. This DSD aesthetic design model is a pre-constructed tooth model.

[0122] It should be noted that the pre-built tooth model can be a texture map model, or it can be other types of aesthetic design models. No specific limitations are imposed on the pre-built tooth model here. The texture of each subsequent frame of the facial expression image is derived from the neutral facial model. The texture coordinates and corresponding UV mappings (texture coordinate mappings) are the same as the neutral facial model of the target user. When the target user's expression changes, the texture also changes. Therefore, calculations and adjustments are performed based on the source texture to obtain the corresponding texture after the expression change.

[0123] In the specific implementation step S104, based on the relative positional relationship information between the intraoral model and the neutral facial model, the part of the intraoral model that has not generated aesthetic design and the pre-built tooth model are superimposed on the neutral facial model that has updated the expression state in each frame, so as to obtain multi-frame expression images (three-dimensional images) that show the effect of aesthetic design.

[0124] It should be noted that the expression state of the neutral facial model is updated by the facial data corresponding to each frame of the image to be processed. Therefore, each frame of the image to be processed can yield a corresponding expression image. This expression image shows a neutral facial model with an expression state, superimposed with the part of the intraoral model that has not generated aesthetic design and the pre-constructed tooth model.

[0125] It should be noted that the intraoral model includes the maxillary model, mandibular model, and occlusal relationship data. During the process of facial expression changes, the mandible should dynamically change its position in accordance with the changes in facial expression. The transformation matrix can be calculated by the position changes of specific facial feature points during the process of facial expression changes. This transformation matrix is ​​implemented as the matrix of mandibular tooth changes. This transformation matrix can then be used to calculate the position of the mandibular model in the neutral facial model that has been updated with the facial expression state.

[0126] In some embodiments, the specific method for obtaining the expression image that demonstrates the aesthetic design effect is as follows: identifying feature point information of a neutral facial model, calculating the positional change of a specific feature point among the feature points, and obtaining a transformation matrix, wherein the specific feature point can be a feature point in the jaw region of the face; and obtaining the updated position of the jaw model in the neutral facial model with the updated expression state based on the relative positional relationship information and the transformation matrix.

[0127] Based on the relative positional relationship information and the updated position, the part of the intraoral model without aesthetic design and the pre-built tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image that shows the effect of the aesthetic design.

[0128] In other embodiments, the portion of the intraoral model for which no aesthetic design has been generated is the intraoral model outside the maxillary anterior region, and the pre-constructed tooth model is the tooth model of the maxillary anterior region.

[0129] To provide a better presentation for target users, 3D animations can be generated and played, thus offering more diverse presentation options.

[0130] In some embodiments, a 3D animation is generated, which includes: an animation with aesthetic design effects consisting of multiple frames of facial expression images (animation with DSD), and / or an animation without aesthetic design effects consisting of multiple frames of images to be processed (animation without DSD).

[0131] If the generated 3D animation contains an animation with aesthetic design effects composed of multiple frames of facial expression images, then playing the 3D animation can dynamically simulate the aesthetic restoration effect of the target user under different facial expression changes.

[0132] For example, playing three frames of facial expression images from the 3D animation shown in Figure 13 can dynamically simulate the aesthetic restoration effect of the target user under different facial expression changes.

[0133] In an animation with aesthetic design effects composed of multiple frames of facial expression images, the facial data corresponding to each frame of the image to be processed is used sequentially from the starting frame to the ending frame to update the facial expression state of the neutral facial model. The parts without aesthetic design effects and the pre-built tooth model are then superimposed to achieve a dynamic simulation of the aesthetic restoration effect of the target user under different facial expression changes. The target user can use this animation to view the aesthetic restoration effect under different facial expression changes from multiple angles and simulate and show the overall effect after aesthetic design in advance.

[0134] In this embodiment, the expression state of a neutral facial model is updated. The portion of the intraoral model without aesthetic design and a pre-built dental model are overlaid onto the updated neutral facial model to obtain an expression image constituting a 3D animation. The 3D animation is played to dynamically demonstrate the aesthetic restoration effects under different expressions to the target user, improving the presentation of the aesthetic restoration effects and the user experience.

[0135] Corresponding to the image processing method provided in the above-described embodiments of this disclosure, referring to FIG14, this disclosure also provides a structural block diagram of an image processing apparatus, which includes: an acquisition unit 100, a forming unit 200, an updating unit 300, and an overlay unit 400;

[0136] The acquisition unit 100 is configured to acquire the relative positional relationship information between the intraoral model and the neutral facial model of the target user, wherein the neutral facial model is a three-dimensional facial model obtained by scanning the target user in a non-expression state.

[0137] Forming unit 200 is configured to use base data of basic expressions to form multiple facial data corresponding to the expression state. The base data of basic expressions is constructed based on a neutral facial model.

[0138] Update unit 300 is configured to update the expression state of the neutral facial model multiple times using facial data.

[0139] The overlay unit 400 is configured to overlay the portion of the intraoral model without aesthetic design and the pre-built tooth model onto a neutral facial model that has been updated with an expression state, based on relative positional relationship information, to obtain an expression image that demonstrates the effect of the aesthetic design.

[0140] In some embodiments, the portion of the intraoral model for which no aesthetic design has been generated is the intraoral model outside the maxillary anterior region, and the pre-constructed tooth model is the tooth model of the maxillary anterior region.

[0141] In this embodiment, the expression state of a neutral facial model is updated. The portion of the intraoral model without aesthetic design and a pre-built dental model are overlaid onto the updated neutral facial model to obtain an expression image constituting a 3D animation. The 3D animation is played to dynamically demonstrate the aesthetic restoration effects under different expressions to the target user, improving the presentation of the aesthetic restoration effects and the user experience.

[0142] Preferably, as shown in Figure 14, the intraoral model includes a maxillary model, a mandibular model, and occlusal relationship data. The overlay unit 400 includes an identification subunit, a determination subunit, and an overlay subunit. The execution principle of each subunit is as follows:

[0143] The recognition subunit is configured to recognize feature point information of a neutral facial model, calculate the positional changes of specific feature points, and obtain a transformation matrix.

[0144] The determined sub-units are configured to obtain the updated position of the mandibular model in the neutral facial model with updated expression states based on relative positional relationship information and transformation matrix.

[0145] The overlay subunit is configured to overlay the portion of the intraoral model without aesthetic design and the pre-built tooth model onto a neutral facial model that has been updated with an expression state, based on relative positional relationship information and update position, to obtain an expression image that demonstrates the effect of the aesthetic design.

[0146] Preferably, in conjunction with the content shown in Figure 14, the forming unit 200 is specifically configured to: form multiple facial data corresponding to the expression state using weight parameters of basic expressions related to the expression change images of other users, wherein the expression change images of other users are obtained by capturing the expression change of other users.

[0147] Preferably, as shown in Figure 14, the forming unit 200 includes an imaging subunit and a forming subunit, and the execution principle of each subunit is as follows:

[0148] The shooting subunit is configured to capture the facial expression changes of the target user to obtain multiple frames of images to be processed.

[0149] The forming subunit is configured to use base data of multiple frames of images to be processed and basic expressions to form multiple facial data corresponding to the multiple frames of images to be processed.

[0150] In specific implementation, this forming subunit is configured to: determine the basic facial expressions associated with the multiple frames of images to be processed, and determine the weight parameters of the basic facial expressions associated with the multiple frames of images to be processed. The multiple frames of images to be processed contain RGB data and depth data. Using the basic data and weight parameters of the basic facial expressions associated with the multiple frames of images to be processed, multiple facial data corresponding to the multiple frames of images to be processed are formed.

[0151] Accordingly, the image processing apparatus also includes:

[0152] The generation unit is configured to generate a 3D animation, which includes: an animation with aesthetic design effects consisting of multiple frames of facial expression images, and / or an animation without aesthetic design effects consisting of multiple frames of images to be processed.

[0153] Preferably, in conjunction with the content shown in FIG14, the image processing apparatus further includes:

[0154] The design unit is configured to generate a pre-built dental model for the target user using aesthetic design software.

[0155] Preferably, referring to the content shown in Figure 14, the acquisition unit 100 includes an acquisition subunit and an alignment subunit, and the execution principle of each subunit is as follows:

[0156] The acquisition sub-unit is configured to acquire the target user's smiling face model, intraoral model, and neutral face model. The smiling face model is a 3D face model obtained by scanning the target user in a toothy smiling expression state.

[0157] The alignment subunit is configured to align the intraoral model to the smiling face model and the neutral face model to the smiling face model to obtain the relative positional relationship information between the intraoral model and the neutral face model.

[0158] Preferably, referring to the content shown in Figure 14, the forming unit 200 includes a processing subunit, an identification subunit, and a construction subunit. The execution principle of each subunit is as follows:

[0159] The processing sub-unit is configured to process the mesh structure at the lips of the neutral facial model in order to separate the upper and lower lips of the neutral facial model.

[0160] The recognition subunit is configured to recognize feature point information of the processed neutral facial model.

[0161] The sub-units are configured to construct base data for multiple basic expressions through deformation transfer based on the processed neutral facial model and feature point information.

[0162] Preferably, this disclosure also provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a communication bus; wherein, the processor is configured to call and execute a program stored in the memory; the memory is configured to store a program, the program being configured to implement the image processing method disclosed in the above method embodiments.

[0163] Preferably, this disclosure also provides a computer-readable storage medium storing computer-executable instructions configured to perform the image processing method disclosed in the above-described method embodiments.

[0164] Preferably, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the method of any of the foregoing embodiments.

[0165] In summary, the embodiments of this disclosure provide an image processing method, apparatus, electronic device, and storage medium for updating the expression state of a neutral facial model. The portion of the intraoral model without aesthetic design and a pre-constructed dental model are superimposed onto the updated expression state of the neutral facial model, thereby obtaining an expression image showcasing the aesthetic design effect, improving the display effect of aesthetic restoration and the user experience.

[0166] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0167] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0168] The above description of the disclosed embodiments enables those skilled in the art to make or use this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Industrial applicability

[0169] The image processing solution provided in this disclosure updates the expression state of a neutral facial model, overlays the parts of the intraoral model that have not generated aesthetic design and the pre-built tooth model onto the neutral facial model whose expression state has been updated, thereby obtaining an expression image that displays the aesthetic design effect, improving the display effect of aesthetic restoration and user experience, and has strong industrial applicability.

Claims

1. An image processing method, wherein, The method includes: The relative positional relationship information between the intraoral model and the neutral facial model of the target user is obtained, wherein the neutral facial model is a three-dimensional facial model obtained by scanning the target user in a non-expression state; Using the base data of basic expressions, multiple facial data corresponding to the expression state are formed. The base data of basic expressions is constructed based on the neutral facial model. The facial data is used to update the expression state of the neutral facial model multiple times; Based on the relative positional relationship information, the part of the intraoral model without aesthetic design and the pre-constructed tooth model are superimposed on the neutral facial model with updated expression state to obtain an expression image that shows the effect of aesthetic design.

2. The method of claim 1, wherein, The intraoral model includes a maxillary model, a mandibular model, and occlusal relationship data; Based on the relative positional relationship information, the portion of the intraoral model without aesthetic design and the pre-constructed tooth model are superimposed onto the neutral facial model whose expression state has been updated, to obtain an expression image demonstrating the aesthetic design effect, including: The feature point information of the neutral face model is identified, and the positional changes of specific feature points among the feature points are calculated to obtain the transformation matrix. Based on the relative positional relationship information and the transformation matrix, the updated position of the mandibular model in the neutral facial model with the updated expression state is obtained; Based on the relative positional relationship information and the updated position, the portion of the intraoral model without aesthetic design and the pre-built tooth model are superimposed onto the neutral facial model with updated expression state to obtain an expression image that demonstrates the effect of the aesthetic design.

3. The method of claim 1, wherein, Using the base data of basic facial expressions, multiple facial data points corresponding to the expression states are generated, including: Capture the target user's facial expressions to obtain multiple frames of images to be processed; Using the base data of the multiple frames of images to be processed and the basic facial expressions, multiple facial data corresponding to the multiple frames of images to be processed are formed.

4. The method of claim 3, wherein, Using the base data of the multiple frames of images to be processed and the basic facial expressions, multiple facial data corresponding to the multiple frames of images to be processed are formed, including: Based on the multi-frame images to be processed, determine the basic facial expressions associated with the multi-frame images to be processed, and determine the weight parameters of the basic facial expressions associated with the multi-frame images to be processed. The multi-frame images to be processed contain RGB data and depth data. Using the base data and weight parameters of the basic expressions associated with the multi-frame images to be processed, multiple facial data corresponding to the multi-frame images to be processed are formed.

5. The method of any one of claims 1-4, wherein, The portion of the intraoral model for which no aesthetic design has been generated is the intraoral model outside the maxillary anterior region, and the pre-constructed tooth model is the tooth model of the maxillary anterior region.

6. The method of any one of claims 1-4, wherein, Before overlaying the unaesthetically designed portions of the intraoral model and the pre-built dental model onto the updated facial expression model, the process further includes: Using aesthetic design software, a dental model designed for the target user is generated as the pre-built dental model.

7. The method of claim 1, wherein, Using the base data of basic facial expressions, multiple facial data points corresponding to the expression states are generated, including: By using weight parameters of basic facial expressions related to the facial expression change images of other users, multiple facial data corresponding to facial expression states are formed, wherein the facial expression change images of other users are obtained by capturing the facial expression change of the other users.

8. The method of claim 3 or 4, wherein, Also includes: Generate a 3D animation, the 3D animation comprising: an animation with aesthetic design effects composed of multiple frames of the facial expression images, and / or an animation without aesthetic design effects composed of the multiple frames of the images to be processed.

9. The method of claim 1, wherein, Obtain the relative positional relationship information between the target user's intraoral model and neutral facial model, including: Acquire a smiling facial model, an intraoral model, and a neutral facial model of the target user, wherein the smiling facial model is a three-dimensional facial model obtained by scanning the target user in a toothy smiling expression state; The intraoral model is aligned to the smiling face model, and the neutral face model is aligned to the smiling face model to obtain the relative positional relationship information between the intraoral model and the neutral face model.

10. The method of any one of claims 1-4, wherein, The process of constructing base data for basic facial expressions based on the neutral facial model includes: The mesh structure at the lips of the neutral face model is processed to separate the upper and lower lips of the neutral face model; Feature point information of the neutral face model after identification processing; Based on the processed neutral facial model and the feature point information, base data for multiple basic expressions are constructed through deformation transfer.

11. An image processing apparatus, comprising: The device includes: The acquisition unit is configured to acquire relative positional relationship information between the intraoral model and the neutral facial model of the target user, wherein the neutral facial model is a three-dimensional facial model obtained by scanning the target user in a non-expression state. The forming unit uses the base data of basic expressions to form multiple facial data corresponding to the expression state. The base data of basic expressions is constructed based on the neutral facial model. The update unit is configured to update the expression state of the neutral facial model multiple times using the facial data; The overlay unit is configured to overlay the portion of the intraoral model without aesthetic design and the pre-built tooth model onto the neutral facial model with updated expression state, based on the relative positional relationship information, to obtain an expression image that displays the effect of the aesthetic design.

12. An electronic device, comprising: include: A processor and a memory are connected via a communication bus; wherein the processor is configured to call and execute a program stored in the memory; The memory is configured to store a program that is configured to implement the image processing method as described in any one of claims 1-10.

13. A computer readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions configured to perform the image processing method according to any one of claims 1-10.

14. A computer program, wherein, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-10.

Images