Detecting tooth attachments and other auxiliaries using 2d images

Abstract

A method for detecting tooth attachments and other dental auxiliaries using 2D images is provided. The method includes obtaining a 2D dentition image that includes a depiction of a patient's dentition; obtaining a 3D dentition model that includes a depiction of the patient's dentition, the 3D dentition model indicating one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliary components are present in the 2D dentition image based at least in part on information from the 3D dentition model; and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

Description

Attorney Docket No.: 28510.983 (L0820PCT)DETECTING TOOTH ATTACHMENTSAND OTHER AUXILIARIES USING 2D IMAGESTECHNICAL FIELD

[0001] The instant specification generally relates to systems and methods for detecting dental components, and in particular to systems and methods for detecting tooth attachments and other auxiliaries using two-dimensional images.BACKGROUND

[0002] Dental auxiliary components (also referred to as orthodontic objects), such as attachments or buttons, help patients achieve desired outcomes for orthodontic treatments. Such components are bonded to specific teeth during treatment. The components serve as anchor points for other dental treatment components or devices, such as orthodontic aligners, that move teeth into target positions.SUMMARY

[0003] The below summary is a simplified summary of the disclosure in order to provide a basic understanding of some aspects of the disclosure. This summary is notan extensive overview of the disclosure. It is intended neither to identify key or critical elements of the disclosure, nor delineate any scope of the particular embodiments of the disclosure or any scope of the claims. Its sole purpose is to present some concepts of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] In a first example implementation, a method for detecting orthodontic components includes: obtaining a two-dimensional (2D) dentition image that includes a depiction of a patient's dentition; obtaining a three-dimensional (3D) dentition model that includes a depiction of the patient's dentition, the 3D dentition model indicating one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliary components are present in the 2D dentition image based at least in parton information from the 3D dentition model; and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

[0005] In a second example implementation, a system for detecting orthodontic components includes: one or more processors and a memory coupled to the one or more processors that stores computer program instructions that, when executed by the one or more processors, perform a computer-implemented method that includes obtaining a 2D dentition image that includes a depiction of a patient's dentition; obtaining a 3D dentition model that includes a depiction of the patient's dentition, the 3D dentition model indicating one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliaryAttorney Docket No.: 28510.983 (L0820PCT)components are present in the 2D dentition image based at least in parton information from the 3D dentition model; and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

[0006] In a third example implementation, a computer-readable storage medium for detecting orthodontic components includes instructions, when executed by one or more processors, performs a computer-implemented method that includes obtaining a 2D dentition image that includes a depiction of a patient's dentition; obtaining a 3D dentition model that includes a depiction of the patient's dentition, the 3D dentition model indicating one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliary components are present in the 2D dentition image based at least in parton information from the 3D dentition model; and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

[0007] In a fourth example implementation, a method fortraining a model for detecting orthodontic components includes: obtaining a 2D dentition image including a depiction of a patient's dentition and one or more first auxiliary components; obtaining a 3D dentition model including a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more second auxiliary components; registering the 3D dentition model to the 2D dentition image; corresponding the one or more second auxiliary components of the 3D dentition model with the one or more first auxiliary components of the 2D dentition image based at least in parton information from the 3D dentition model; generating a training dataset item including a training input including the 2D dentition image and a target output including one or more locations of the one or more first auxiliary components of the 2D dentition image; and training an artificial intelligence (Al) model on a training dataset, wherein the training dataset includes the training dataset item, and wherein the Al model is trained to identify auxiliary components in input 2D dentition images.

[0008] In a fifth example implementation, a system for detecting orthodontic components includes a first computing device configured to: receive a 2D dentition image including a depiction of a patient's dentition; receive a 3D dentition model including a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components; register the 3D dentition model to the 2D dentition image; determine whether the one or more expected auxiliary components are present in the 2D dentition image based at least in part on information from the 3D dentition model; and generate an indication that a first expected auxiliary component of the one or more expected auxiliary components is missing from the 2D dentition image.

[0009] In a sixth example implementation, a method for detecting orthodontic components includes: obtaining a 2D dentition image including a depiction of a patient's dentition; processing the 2DAttorney Docket No.: 28510.983 (L0820PCT)dentition image using one or more first models, wherein the one or more first models output information identifying locations of a plurality of teeth in the 2D dentition image; generating one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth in the 2D dentition image, each cropped 2D image of the one or more cropped 2D images associated with a tooth of the plurality of teeth and including pixels of the 2D dentition image for the tooth and excluding pixels of the 2D dentition image for a remainder of the plurality of teeth; processing the one or more cropped 2D images using one or more second models, wherein for each processed cropped 2D image the one or more second models output an indication of whether the tooth associated with the cropped 2D image has one or more attached auxiliary components; and outputting an indication of which teeth have attached auxiliary components.

[0010] In a seventh example implementation, a method for training a model for detecting orthodontic components includes: obtaining a 2D dentition image including a depiction of a patient's dentition; processing the 2D dentition image using one or more first models, wherein the one or more first models output information identifying locations of a plurality of teeth in the 2D dentition image; generating one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth in the 2D dentition image, each cropped 2D image of the one or more cropped 2D images associated with a tooth of the plurality of teeth and including pixels of the 2D dentition image for the tooth and excluding pixels of the 2D dentition image for a remainder of the plurality of teeth; determining, for each cropped 2D image of the one or more cropped 2D images, a label specifying whether the cropped 2D image includes an auxiliary component; and training one or more second models to process an input of a cropped 2D image of a tooth and to output an indication of whether the tooth in the cropped 2D image has one or more attached auxiliary components based on the one or more cropped 2D images and associated labels.

[0011] In an eighth example implementation, a method for detecting orthodontic components includes: obtaining a 2D dentition image including a depiction of a patient's dentition; providing the 2D dentition image to a system configured to process the 2D dentition image using one or more first models, wherein the one or more first models output information identifying locations of a plurality of teeth in the 2D dentition image, generate one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth in the 2D dentition image, each cropped 2D image of the one or more cropped 2D images associated with a tooth of the plurality of teeth and including pixels of the 2D dentition image for the tooth and excluding pixels of the 2D dentition image for a remainder of the plurality of teeth, and process the one or more cropped 2D images using one or more second models, wherein for each processed cropped 2D image the one or more second models output an indication of whether the tooth associated with the cropped 2D imageAttorney Docket No.: 28510.983 (L0820PCT)has one or more attached auxiliary components; receiving, from the system, the indication of which teeth have attached auxiliary components; and outputting the indication on a user interface.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects and embodiments of the present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various aspects and embodiments of the disclosure, which, however, should not be taken to limit the disclosure to the specific aspects or embodiments, but are for explanation and understanding only.

[0013] FIG. 1 A illustrates a flow diagram of an example method for detecting tooth attachments and other auxiliaries using two-dimensional (2D) images, in accordance with some embodiments of the present disclosure.

[0014] FIG. 1 B illustrates a flow diagram of an example method for training a model for detecting tooth attachments and other auxiliaries using 2D images, in accordance with some embodiments of the present disclosure.

[0015] FIG. 1C illustrates a flow diagram of an example method for detecting auxiliary components using per-tooth processing, in accordance with some embodiments of the present disclosure.

[0016] FIG. 1D illustrates a flow diagram of an example method for detecting auxiliary components using a three-dimensional dentition model, in accordance with some embodiments of the present disclosure.

[0017] FIG. 1E illustrates a flow diagram of an example method for selecting classification models based on treatment plan information, in accordance with some embodiments of the present disclosure.

[0018] FIG. 1F illustrates a flow diagram of an example method for determining types of auxiliary components present at a tooth using multiple classification models, in accordance with some embodiments of the present disclosure.

[0019] FIG. 1G illustrates a flow diagram of an example method for training a model for detecting auxiliary components using per-tooth processing, in accordance with some embodiments of the present disclosure.

[0020] FIG.2A illustrates an example 2D dentition image, in accordance with some embodiments of the present disclosure.

[0021] FIG.2B illustrates the example 2D dentition image of FIG.2A with bounding boxes surrounding some auxiliary components, in accordance with some embodiments of the present disclosure.Attorney Docket No.: 28510.983 (L0820PCT)

[0022] FIG.2C illustrates an example 2D dentition image showing a frontal view of a patient's dentition, in accordance with some embodiments of the present disclosure.

[0023] FIG.2D illustrates an example 2D dentition image with bounding boxes indicating detected or expected locations of auxiliary components, in accordance with some embodiments of the present disclosure.

[0024] FIG.2E illustrates an example segmented dentition image showing individual teeth identified through segmentation, in accordance with some embodiments of the present disclosure.

[0025] FIG.2F illustrates an example cropped 2D dentition image comprising a depiction of a single tooth from a patient's dentition, in accordance with some embodiments of the present disclosure.

[0026] FIG.2G illustrates the example 2D dentition image of FIG.2F with bounding boxes identifying regions of interest associated with auxiliary components, in accordance with some embodiments of the present disclosure.

[0027] FIG.2H illustrates an example cropped 2D image with masked regions, in accordance with some embodiments of the present disclosure.

[0028] FIG.21 illustrates an example cropped 2D image comprising a depiction of a single auxiliary component, in accordance with some embodiments of the present disclosure.

[0029] FIG. 3 illustrates an example three-dimensional (3D) dentition model, in accordance with some embodiments of the present disclosure.

[0030] FIG.4A illustrates a model training workflow and a model application workflow for a dental treatment computing system capable of detecting tooth attachments and other auxiliaries using 2D images, in accordance with an embodiment of the present disclosure.

[0031] FIG.4B illustrates a model training workflow and a model application workflow for a dental treatment computing system capable of detecting tooth attachments and other auxiliaries using pertooth processing, in accordance with an embodiment of the present disclosure.

[0032] FIG. 5 illustrates an example system architecture capable of detecting tooth attachments and other auxiliaries using 2D images, in accordance with some embodiments of the present disclosure.

[0033] FIG. 6 depicts a block diagram of an example processing device operating in accordance with one or more embodiments of the present disclosure.

[0034] FIG. 7A illustrates an exemplary tooth repositioning dental appliance or aligner that can be worn by a patient in order to achieve an incremental repositioning of individual teeth in the jaw, in accordance with an embodiment of the present disclosure.

[0035] FIG. 7B illustrates a tooth repositioning system including a plurality of appliances, in accordance with some embodiments of the present disclosure.Attorney Docket No.: 28510.983 (L0820PCT)

[0036] FIG.7C illustrates a method of orthodontic treatment using a plurality of appliances, in accordance with some embodiments of the present disclosure.

[0037] FIG.8 illustrates a method for designing an orthodontic appliance to be produced by direct fabrication, in accordance with embodiments of the disclosure.

[0038] FIG.9 illustrates a method for digitally planning an orthodontic treatment and / or design or fabrication of an appliance, in accordance with embodiments of the disclosure.DETAILED DESCRIPTION

[0039] Dental auxiliary components (referred to, herein, as "auxiliary components") refers to objects that are affixed to one or more teeth to engage a dental appliance (e.g., an aligner, palatal expander, retainer, mouth guard, orthodontic wires) to facilitate an orthodontic treatment, such as dental attachments, buttons, power arms, brackets, etc. In some embodiments, auxiliary components may also include components that are attached to and integrated into a dental appliance (e.g., such as power ridges). These auxiliary components are used in dental treatments to help patients achieve desired outcomes for dental treatments, such as orthodontic treatments, palatal expansion treatments, and / or retention. Such components may be bonded to specific teeth during treatment. The components may serve as anchor points for other dental treatment devices, such as orthodontic aligners that move teeth into target positions, palatal expanders that expand the palate to target positions, and so on.

[0040] Sometimes, an auxiliary component may become prematurely damaged or debonded from the patient's teeth (e.g., through ordinary wear and tear, unintended forces, defective bonding, etc.) during the patient's dental treatment. These auxiliary components may be necessary for effective treatment, and continuing treatment without them may not produce desired treatment outcomes. For example, a tooth attachment may be required for proper retention of a dental appliance (e.g., an aligner, a palatal expander, a retainer) or for accomplishing a desired tooth movement. Not having this attachment can compromise treatment with the dental appliance. A patient may take a photograph of the patient's mouth and send the photographs to a dental professional so the dental professional can check if any of the patient's auxiliary components are missing. However, auxiliary components can be difficult to identify in the photographs submitted to the dental professional because the auxiliary components are often colored to match the color of the patient's teeth, the photographs may be blurry, or auxiliary components may be occluded by other objects in the photographs (e.g., the patient's teeth, inner cheek). A dental treatment software system may include software that can perform image recognition on the photographs to identify auxiliary components. However, such software that relies on the photographs as input may provide inaccurate results because of the limited visual information the photographs provide.Attorney Docket No.: 28510.983 (L0820PCT)

[0041] Automated detection of auxiliary components is invaluable in helping doctors and orthodontists identify when auxiliary components fall out during treatment, without the need for in-person visits. However, auxiliary components are often difficult to detect from images, as colors are often chosen to be close to the patient's tooth color, images are sometimes dark or blurry, and auxiliary components can sometimes be small or even partially occluded by other objects in the image. Using full image resolution would provide the most granular detail for detecting these difficult-to-see objects. However, training and inference on very large full images might not be practical. A second difficulty is classifying detected auxiliary components according to their type (e.g., attachments versus buttons) and also according to which tooth they are bonded to. Both of these items could be partially alleviated by performing training and inference on a single tooth ata time. This could be done, for instance, by cropping the image to a particular tooth using a tooth segmentation of the image. By focusing on a single tooth, the image resolution can be higher, and the tooth number is already known. Additionally, labeling attachment versus button is simplified since it can be performed on single teeth at a time, instead of the whole image of multiple teeth, allowing for efficient creation of training data.

[0042] Detection of dental features for a patient using some embodiments may involve taking an image of a patient's entire dentition, feeding the whole image into an object detection algorithm, and then assigning detected objects to a particular tooth for reporting to a dental professional. The detection may be for any dental feature of relevance, such as dental caries, gingival recession, or auxiliary components such as attachments and buttons. In some cases, this process may suffer from drawbacks. First, the original resolution of full dentition images may be too large for many machine learning algorithms due to algorithmic or computational constraints. As such, images may need to be resized smaller prior to consumption by a machine learning model. This resizing may lead to a loss of information, making the detection of certain subtle features difficult and reducing the sensitivity of the model for detection. Second, after identification, the features may still need to be assigned to individual teeth for reporting to a dental professional. This assignment may be imperfect if the feature is not centered entirely within the boundary of the tooth, such as for attachments, buttons, and gingival recession. In some aspects, these drawbacks may be minimized by using per-tooth processing methodology, which may be applied to orthodontic object detection. Orthodontic objects may include attachments and buttons, among other auxiliary components.

[0043] Aspects and implementations of the present disclosure address the above and other challenges by providing systems and methods for determining whether an auxiliary component is present in a two-dimensional (2D) dentition image (e.g., a photograph submitted by a patient). In a first aspect, the systems and methods register a three-dimensional (3D) dentition model of the patient's dentition to the 2D dentition image and determine, from a comparison of the expected auxiliaryAttorney Docket No.: 28510.983 (L0820PCT)components from the registered 3D dentition model and the auxiliary components detected from the 2D dentition image, whether an auxiliary component is missing. In a second aspect, the systems and methods perform per-tooth auxiliary component detection by segmenting the 2D dentition image to identify individual teeth, generating cropped 2D images for each tooth, and processing the cropped 2D images using one or more classification models to determine whether auxiliary components are present on each tooth. In a third aspect, the systems and methods combine the first and second aspects by using 3D dentition model registration to identify expected locations of auxiliary components and then generating cropped 2D images based on those expected locations for per-auxiliary-component classification. If an auxiliary component is missing, the systems and methods may output an indication on a user interface (Ul) notifying a user of the III (e.g., a dental professional or the patient) that the auxiliary component is missing. The patient may then go to the dental professional and have the missing auxiliary component replaced in some instances.

[0044] In one embodiment, the 2D dentition image may depict one or more auxiliary components located on the patient's dentition, and the systems and methods may match one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image. In some embodiments, the 3D dentition model can indicate one or more expected auxiliary components on respective teeth of the patient at respective target locations, and the systems and methods can determine one or more locations of the auxiliary components in the 2D dentition image and compare the locations of the auxiliary components in the 2D dentition image with the target locations. The systems and methods can determine that an expected auxiliary component is missing from the 2D dentition image by determining that a target location at which an auxiliary should be located does not correspond to a location of the auxiliary components detected in the 2D dentition images.

[0045] In some embodiments, an artificial intelligence (Al) model, such as a machine learning model, may use the 2D dentition image as input and may identify auxiliary components in the 2D dentition image. The Al model may indicate a location for each auxiliary component in the 2D dentition image. The Al model can output a bounding box for each auxiliary component, tooth labels indicating which respective tooth a respective auxiliary component is attached to, semantic segmentation or instance segmentation information identifying individual teeth and / or auxiliary components, and / or a component type (e.g., orthodontic button or orthodontic attachment) of one or more detected auxiliary components.

[0046] In some embodiments, the systems and methods can generate a binary mask based on the registered 3D dentition model. The binary mask can include data indicating one or more locations in the 2D dentition image corresponding to the expected auxiliary components. The binary mask can thenAttorney Docket No.: 28510.983 (L0820PCT)be used to modify the 2D dentition image so that when an Al model analyzes the 2D dentition image to identify locations of auxiliary components, the Al model can focus on areas designated by the binary mask and increase the Al model's accuracy and efficiency.

[0047] In embodiments, the systems and methods can generate training dataset items used to train the Al model to detect auxiliary components in the 2D dentition image. An item of the training dataset may have a training inputthat includes a 2D dentition image and a corresponding target output that includes locations of auxiliary components in the training input and tooth labels for the locations. The locations and / or tooth labels can be derived from registering a 3D dentition model to the 2D dentition image of the training input. In some embodiments, the systems and methods may use a registered 3D dentition model to assist in identifying mislabeled training data of a training dataset for the Al model and remove or modify the mislabeled item.

[0048] In some embodiments, the systems and methods perform per-tooth auxiliary component detection. The systems and methods may process the 2D dentition image using one or more first models that output information identifying locations ofa plurality of teeth in the 2D dentition image. The one or more first models may include an Al model trained to perform object detection to detect teeth, wherein the Al model outputs bounding boxes around each of the plurality of teeth in the 2D dentition image. Alternatively or additionally, the one or more first models may include an Al model trained to perform segmentation of teeth in images, wherein the Al model outputs segmentation information for the plurality of teeth in the 2D dentition image. The systems and methods may generate one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth, where each cropped 2D image is associated with a tooth and includes pixels of the 2D dentition image for the tooth while excluding some or all pixels for a remainder of the plurality of teeth. In some embodiments, the systems and methods may identify, for a cropped 2D image and based on the segmentation information, one or more pixels of the cropped 2D image that do not belong to the tooth associated with the cropped 2D image, and may modify the cropped 2D image by setting values for those pixels to a predetermined value (e.g., black), wherein the pixels having the predetermined value are ignored by one or more second models used for classification.

[0049] In some embodiments, the systems and methods process the one or morecropped 2D images using one or more second models, wherein for each processed cropped 2D image the one or more second models output an indication of whether the tooth associated with the cropped 2D image has one or more attached auxiliary components. The one or more second models may include one or more classification models trained to perform image classification. In some embodiments, the one or more classification models may include a plurality of classification models, each trained to perform classification for a different type of auxiliary component, such as a first modelAttorney Docket No.: 28510.983 (L0820PCT)that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and / or a fourth model that identifies brackets. In some embodiments, the systems and methods may determine, based on a treatment plan, one or more types of expected auxiliary components for a tooth, and may determine which classification models of the plurality of classification models to apply to the cropped 2D image based on the one or more types of expected auxiliary components, wherein the cropped 2D image is processed using the determined classification models and is not processed by a remainder of the plurality of classification models.

[0050] In some embodiments, an output of the plurality of classification models with reference to a cropped 2D image may include a plurality of confidence values, each associated with a different type of auxiliary component. The systems and methods may determine a type of auxiliary component that is present at the tooth associated with the cropped 2D image based on determining that the confidence value associated with the type of auxiliary component satisfies one or more criteria. The one or more criteria may include a first criterion that a confidence value associated with the type of auxiliary component is greater than confidence values associated with a remainder of types of auxiliary components, and / or a second criterion that the confidence value associated with the type of auxiliary component exceeds a confidence threshold. In some embodiments, different confidence thresholds may be applied to different types of auxiliary components based on the rarity or prior probability of the auxiliary component type. For example, buttons may be considered rare events compared to attachments, and the confidence threshold for confirming the presence of a button may be set higher (e.g., 0.9) than the confidence threshold for confirming the presence of an attachment (e.g., 0.5). This approach may reduce false positives for rare auxiliary component types while maintaining sensitivity for more common auxiliary component types. In some embodiments, the systems and methods may determine that the one or more criteria are satisfied by the confidence values associated with multiple types of auxiliary components, and may determine whether each of the multiple types of auxiliary components are present at the tooth. This determination may include determining that a rule indicates that the multiple types of auxiliary components cannot coexist on the tooth, and selecting the type of auxiliary component of the multiple types of auxiliary components that satisfies a selection rule, such as a rule to select a type of auxiliary component associated with a highest confidence value.

[0051] In some embodiments, the systems and methods combine 3D dentition model registration with per-tooth or per-auxiliary-component processing. The systems and methods may obtain a 3D dentition model including a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components, and may register the 3D dentition model to the 2D dentition image. The systems and methods may determine expected locations of the one or more expected auxiliary components based at least in part on information from the 3D dentition model. AAttorney Docket No.: 28510.983 (L0820PCT)cropped 2D image may be associated with a tooth and an expected auxiliary component on the tooth, and the cropped 2D image may include pixels of the 2D dentition image for the associated expected auxiliary component and exclude at least some pixels of the 2D dentition image for the tooth. The cropped 2D image may have a size that is larger than a size of the expected auxiliary component in the 2D dentition image by up to a threshold number of pixels. In some embodiments, the one or more cropped 2D images may include a first cropped 2D image associated with a tooth and a first expected auxiliary component on the tooth and a second cropped 2D image associated with the tooth and a second expected auxiliary component on the tooth.

[0052] In some embodiments, the systems and methods may perform longitudinal monitoring of auxiliary components. The systems and methods may obtain a new 2D dentition image including a new depiction of the patient's dentition at a second time that is later than a first time at which the 2D dentition image is obtained, process the new 2D dentition image using the one or more first models, generate one or more new cropped 2D images, process the one or more new cropped 2D images using the one or more second models, make a comparison of new indications generated from processing of the new cropped 2D images to indications generated from processing of the cropped 2D images, determine any differences between the new indications and the indications based on the comparison, and output a notice of the differences.

[0053] Some embodiments are described herein with reference to auxiliary components used for orthodontic treatment. However, it should be understood that embodiments described herein with respect to orthodontic auxiliary components also apply to other types of auxiliary components and / or to auxiliary components used for other dental treatments. For example, embodiments described with reference to auxiliary components used for orthodontic treatment may also apply to auxiliary components used for palatal expansion treatment. Additionally, embodiments also apply to detection of auxiliary components that are part of or attached to dental appliances rather than to teeth (e.g., such as some types of power ridges). In such an embodiment, the captured 2D images may be of the patient's dentition with the clear dental appliance being worn rather than of the dentition without the dental appliance being worn. In some embodiments, the systems and methods may be applied to hybrid treatment scenarios where a patient is transitioning from brackets and wires to aligners, or where a patient has different treatment modalities on different arches. For example, a patient may have brackets and wires on one arch (e.g., the lower arch) and aligners with attachments on another arch (e.g., the upper arch). In such cases, the systems and methods may monitor auxiliary components on a subset of teeth or a subset of arches based on the treatment modality applicable to each region. The systems and methods may apply different detection models or different detection parameters to different regions of the dentition based on the treatment modality. For example, a bracket detection model may beAttorney Docket No.: 28510.983 (L0820PCT)applied to teeth undergoing bracket-based treatment while an attachment detection model may be applied to teeth undergoing aligner-based treatment.

[0054] Systems and methods for detecting auxiliary components in image data (e.g., in 2D images) will be illustratively described throughout the specification set forth below (e.g., with respect to FIGS. 1A-9). Furthermore, exemplary embodiments of such systems and methodswill follow throughout the specification set forth below (e.g., with respect to FIGS. 1A-9). Embodiments described herein provide significant advantages with respect to virtual assessment of dental treatment progress. Such embodiments can provide systems and methods that register a 3D dentition model of a patient's dentition to a 2D dentition image of the same patient's dentition to determine whether expected auxiliary components are present in the 2D dentition image. Such registration and determining steps result in more increased accuracy with respect to identification of missing auxiliary components. Using the registered 3D dentition model with the 2D dentition image can also improve other aspects of the systems and methods, such as for improved generation of training data and / or identification of mislabeled training data, which can improve the accuracy of Al models used by the systems and methods that are trained using such training data. Additionally, embodiments that perform per-tooth auxiliary component detection provide advantages by enabling higher resolution analysis of individual teeth, simplifying the classification problem, increasing the confidence of outputs of a classification model, and eliminating the need for separate assignment of detected auxiliary components to teeth. With respect to resolution, the original resolution of full dentition images is often too large for many machine learning algorithms due to algorithmic or computational constraints, requiring images to be resized smaller prior to processing. This resizing leads to a loss of information, making the detection of certain subtle features difficult and reducing the sensitivity of the model for detection. By cropping to individual teeth, the systems and methods can process each tooth at higher effective resolution than would be possible with full-image processing, leading to easier identification of auxiliary components that may be small or tooth-colored. With respect to tooth assignment, when auxiliary components are detected in full dentition images, the detected auxiliary components must be assigned to individual teeth for reporting. This assignment may be imperfect if the auxiliary component is not centered entirely within the boundary of the tooth, such as when an attachment is positioned near the boundary between two adjacent teeth. Per-tooth processing eliminates this assignment problem because each cropped image is already associated with a specific tooth, and any auxiliary components detected in that cropped image are inherently assigned to that tooth. Embodiments that combine 3D dentition model registration with per-tooth or per-auxiliary-component processing provide further advantages by enabling focused analysis on specific locations where auxiliary components are expected, reducing false positives, and enabling multiple auxiliary components on a single tooth to be analyzed separately.Attorney Docket No.: 28510.983 (L0820PCT)Accordingly, orthodontic treatment efficacy is improved and optimized in embodiments. Such improvements are likely to result in increased patient satisfaction as well as reduced costs by reducing the number of patient visits to orthodontic professionals to check if auxiliary components are missing.

[0055] Embodiments are described with reference to detection of auxiliary components that are bonded to teeth. For such embodiments, 2D dentition images are generally captured while a patient is not wearing a dental appliance. Embodiments may also include techniques for detecting additional types of auxiliary components that are disposed on, attached to, or integrated with dental appliances. One example of such an auxiliary component is a power ridge. Other examples of such auxiliary components include hooks for elastics. Power ridges are features that may be formed on or attached to dental appliances (such as aligners) to apply additional force to teeth during orthodontic treatment. Unlike attachments and buttons, which are bonded directly to teeth, power ridges may be part of or attached to the dental appliance itself. In embodiments where auxiliary components such as power ridges that are attached to the dental appliance rather than to the teeth are to be detected, the captured 2D images may be of the patient's dentition with the dental appliance being worn rather than of the dentition without the dental appliance being worn. The techniques described herein otherwise work in the same manner as described, except that the images are of teeth wearing a dental appliance rather than of bare teeth without a dental appliance.

[0056] In some embodiments, the per-tooth processing and detection techniques described herein may be adapted to detect other types of dental objects or conditions beyond auxiliary components. For example, the systems and methods may be configured to detect dental caries (cavities), tooth wear, broken teeth, chipped teeth, cracked teeth, tooth fractures, gingival recession, gingival swelling, gingival inflammation, periodontal pockets, tooth discoloration, tooth staining, plaque accumulation, calculus or tartar buildup, enamel erosion, enamel hypoplasia, dental fluorosis, tooth crowding, tooth spacing, malocclusion, open bite conditions, crossbite conditions, overbite conditions, underbite conditions, tooth rotation, tooth tipping, tooth intrusion, tooth extrusion, root exposure, root resorption, bone loss, and other dental conditions or anomalies. The detection techniques may also be applied to monitoring the condition of dental restorations such as crowns, bridges, veneers, inlays, onlays, fillings, dental implants, implant abutments, and other prosthetic components. Additionally, the techniques may be used to detect oral lesions, ulcers, white patches, red patches, swelling, asymmetry, and other soft tissue conditions.

[0057] The embodiments described herein with reference to detection of auxiliary components may be adapted with minimal additional work to detect these other dental objects or conditions. In some aspects, the primary adaptation may involve training the classification models on different types of dental features rather than on auxiliary components. For example, a classification model may be trainedAttorney Docket No.: 28510.983 (L0820PCT)to identify the presence or absence of dental caries in cropped 2D images of individual teeth, or to identify the presence or severity of gingival recession around individual teeth. The per-tooth cropping approach described herein may provide advantages for detection of these other conditions by enabling higher resolution analysis of individual teeth and by simplifying the classification problem. In some embodiments, multiple classification models may be trained for different dental conditions, and the systems and methods may apply one or more of these models to cropped 2D images based on the type of analysis being performed. For example, a dental professional may request analysis for caries detection, gingival recession assessment, or auxiliary component verification, and the appropriate classification models may be selected and applied accordingly.

[0058] In some embodiments, for certain dental conditions there may be no treatment plan or 3D dentition model available for comparison. For example, when monitoring for dental caries, tooth wear, or gingival recession, there may not be a predefined expected state against which to compare current detections. In such instances, a baseline may be established from an earlier 2D dentition image or 3D dentition model, and later detections may be compared to this baseline to identify changes or progression of conditions over time. For example, the systems and methods may obtain a first 2D dentition image captured at an initial time, process the first 2D dentition image to detect dental conditions present at the initial time, and store the detection results as a baseline. At a later time, the systems and methods may obtain a second 2D dentition image, process the second 2D dentition image to detect dental conditions, and compare the detection results to the baseline to identify any changes. The comparison may identify new conditions that were not present at the initial time, progression or worsening of existing conditions, improvementor resolution of previously detected conditions, or stability of conditions over time. The systems and methods may output a notice of any detected changes to a dental professional or patient.

[0059] In some aspects, the baseline comparison approach may be applied to longitudinal monitoring of various dental conditions. For dental caries, the systems and methods may track the appearance of new carious lesions, the progression of existing lesions from incipient to moderate to severe stages, or the arrest or remineralization of lesions following treatment. For tooth wear, the systems and methods may monitor the progression of attrition, abrasion, erosion, or abfraction over time. For gingival conditions, the systems and methods may track changes in gingival margin position indicative of recession or swelling, changes in gingival color or texture indicative of inflammation, or changes in papilla height indicative of periodontal disease progression. For dental restorations, the systems and methods may monitor for signs of restoration failure such as marginal breakdown, secondary caries, fracture, or debonding.Attorney Docket No.: 28510.983 (L0820PCT)

[0060] In some embodiments, the systems and methods may be configured to detect and monitor conditions related to orthodontic appliances beyond auxiliary components. For example, the systems and methods may detect the presence, position, or condition of orthodontic wires, archwires, ligatures, elastic bands, springs, hooks, stops, or other components of fixed orthodontic appliances. The systems and methods may also detect conditions such as wire breakage, bracket debonding, band loosening, or appliance distortion. For removable appliances such as aligners, retainers, or palatal expanders, the systems and methods may detect signs of appliance wear, cracking, warping, or improper fit. The detection of these conditions may enable early identification of appliance issues that could compromise treatment outcomes.

[0061] In some aspects, the systems and methods may be configured to detect conditions related to dental implants and implant-supported restorations. For example, the systems and methods may monitor for signs of peri-implant mucositis or peri-i mp I antitis, such as soft tissue inflammation, swelling, or recession around implant sites. The systems and methods may also detect conditions affecting implant-supported crowns, bridges, or dentures, such as loosening, fracture, wear, or aesthetic changes. For patients with multiple implants, the per-tooth processing approach may be adapted to perimplantprocessing, enabling focused analysis of each implant site.

[0062] In some embodiments, the systems and methods may be configured to detect and classify different types or severities of detected conditions. For example, for dental caries detection, the classification models may be trained to distinguish between incipient caries, moderate caries, and severe caries, or to distinguish between enamel caries and dentin caries. For gingival recession, the classification models may be trained to classify recession according to established classification systems such as the Miller classification or the Cairo classification. For tooth wear, the classification models may be trained to assess wear severity according to established indices. The classification of condition type or severity may enable more informative reporting to dental professionals and may facilitate treatment planning decisions.

[0063] In some aspects, the detection techniques described herein may be applied to screening and triage applications. For example, the systems and methods may be deployed to analyze 2D dentition images submitted by patients to identify potential dental conditions that may warrant professional evaluation. The systems and methods may generate a report identifying detected conditions and their locations, which may be reviewed by a dental professional to determine whether an in-person examination is recommended. This application may be particularly useful for remote or underserved populations with limited access to dental care, enabling identification of patients who may benefit most from professional dental services.Attorney Docket No.: 28510.983 (L0820PCT)

[0064] FIGS. 1 A-1 G are flow diagrams showing methods of identifying auxiliary components in dentition images, in accordance with embodiments of the present disclosure. The methods may be performed by processing logic that comprises hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), ora combination thereof. In one embodiment, processing logic corresponds to a computing device of a dental treatment computing system. The methods may include one or more actions discussed below in relation to a model training workflow and / or model application workflow of FIGS. 4A-B.

[0065] FIG. 1 A illustrates a flow diagram of a method 100 for detecting auxiliary components, in accordance with embodiments of the present disclosure. At block 110, processing logic obtains a 2D dentition image. The 2D dentition image may include a depiction of a patients dentition. The 2D dentition image may include a depiction of the patients dentition captured during a particular stage of the patients treatment. The 2D dentition image may be similar to the 2D dentition image 200 of FIG.2A or the 2D dentition image 250 of FIG. 2B. For example, the 2D dentition image may include visual features corresponding to one or more auxiliary components. An auxiliary component may include an orthodontic button, orthodontic attachment, orthodontic brackets, etc., as discussed herein. Obtaining the 2D dentition image may include an auxiliary component detector retrieving the 2D dentition image from dentition image(s) of a dental treatment datastore and / or receiving the dentition image(s) from a patient device. As discussed below, the patient device may have used a camera associated with the patient device to capture the 2D dentition image(s) and may have provided the 2D dentition image(s) to a dental treatment computing system, which may have stored the received 2D dentition image in the dental treatment datastore.

[0066] At block 120, processing logic obtains a first 3D dentition model. The first 3D dentition model may include a 3D dentition model that is determined to best match the 2D dentition image (e.g., an expected 3D dentition model that corresponds to the particular stage associated with the 2D dentition image). Processing logic may determine a treatment plan being used to treat the patient and a current stage of dental treatment of the treatment plan. Processing logic may then determine a 3D dentition model associated with a particular stage of dental treatment. The 3D dentition model may include a 3D depiction of the patients dentition for the particular stage of dental treatment. The 3D dentition model may reflect an expected condition (e.g., teeth arrangement) of the patients teeth predicted for the current stage (e.g., as predicted in an initial treatment plan, or otherwise predicted at a timepoint that is prior to the current stage). The 3D dentition model can indicate one or more expected auxiliary components that are called for by the dental treatment plan. The 3D dentition model may be similar to the 3D dentition model 300 of FIG.3, discussed below. For example, the 3D dentition model may include one or more dental features of the patient associated with the 3D dentition model. TheAttorney Docket No.: 28510.983 (L0820PCT)dental features may be the patients teeth or gums. The auxiliary components depicted in the 3D dentition model may be referred to as “expected” because they are expected to be presenton the patients dentition (and, thus, are expected to be present in the 2D dentition image) but may be missing.

[0067] Obtaining the 3D dentition model may include the auxiliary component detector retrieving the 3D dentition model from dentition models of the dental treatment datastore. 3D scanning equipment (e.g., an intraoral scanner) may have been used to capture images of the patients dentition. A dental professional device or other computing devices (e.g., computing devices associated with a dental professional) may generate a corresponding 3D dentition model corresponding to the captured images, and may generate a treatment plan including a sequence of treatment stages each comprising an expected 3D dentition model of a planned arrangement of the patients dentition for the respective treatment stage. The 3D dentition models (the 3D dentition model corresponding to the captured images and / or one or more of the expected 3D dentition models) may be provided to a dental treatment computing system, which may have stored the 3D dentition models in the dental treatment datastore. Alternatively, the dental treatment computing system may have performed one or more of these operations.

[0068] At block 430, processing logic registers the 3D dentition model to the 2D dentition image. Registering the 3D dentition model to the 2D dentition image may include aligning the 3D dentition model with the 2D dentition image. The aligning process can include feature extraction from both the 2D dentition image and the 3D dentition model, such as to identify points, lines, and / or surfaces. These features may be matched between the 3D dentition model and the 2D dentition image based on geometric or appearance-based similarities. Once matched, a transformation matrix is generated and used to align the 3D dentition model’s coordinate system with the 2D dentition image’s coordinate system. The transformation can involve rotation, translation, and / or scaling operations. Iterative optimization techniques may refine the transformation matrix and minimize an error between the projected 3D dentition model features and the corresponding 2D dentition image features. Any suitable 3D to 2D registration algorithm can be used, for example, as described in U.S. Patent Nos. 11,020,205 and 11 ,723,748, which are incorporated by reference herein in their entirety.

[0069] In some embodiments, based on the registration between the 2D dentition image(s) and the 3D dentition model, processing logic may determine locations in the 2D dentition image(s) at which auxiliary components are expected (e.g., by mapping the shapes / locations / sizes of the auxiliary components from the 3D dentition model to the 2D dentition image(s)).

[0070] At block 140, processing logic determines whether the one or more expected auxiliary components are present in the 2D dentition image. The determining may be based, at least in part, on information from the 3D dentition model. In one embodiment, the auxiliary component detector mayAttorney Docket No.: 28510.983 (L0820PCT)match one or more of the expected auxiliary components of the registered 3D dentition model to one or more auxiliary components of the 2D dentition image.

[0071] The auxiliary component detector may determine one or more locations of the auxiliary components in the 2D dentition image. In one embodiment, the auxiliary component detector may use an Al model to determine the locations of one or more auxiliary components depicted in the 2D dentition image. The Al model may include an auxiliary component detection model, discussed herein, which may include an image recognition Al model trained and configured to detect auxiliary components in an input image. The auxiliary component detection model may use the 2D dentition image as input and may perform an inference calculation. The auxiliary component detection model can generate output data that indicates one or more locations of auxiliary components in the input 2D dentition image. The output data may include data indicating a bounding box around each identified auxiliary component and / or a segmentation map indicating pixels ofone or more dental auxiliary components. In some embodiments, expected locations of dental auxiliary components are input into the Al model together with the 2D dentition image. In some embodiments, the Al model may be trained to focus on locations at which auxiliary components are expected (e.g., from input expected location information), and to output for each such location whether an auxiliary component 204 was detected. In some embodiments, the Al model may be configured to have increased sensitivity at locations where auxiliary components are expected based on the registered 3D dentition model. For example, the Al model may apply a lower detection threshold or increased attention weights at expected auxiliary component locations, thereby increasing the likelihood of detecting auxiliary components that are present at those locations while maintaining a higher threshold at other locations to reduce false positives.

[0072] The auxiliary component detector may obtain one or more target locations of the expected auxiliary components of the registered 3D dentition model. The registered 3D dentition model may include data indicating a location of each expected auxiliary component. The location data may include a coordinate in a 3D coordinate system used by the 3D dentition model projected onto the 2D dentition image. The auxiliary component detector may match the one or more expected auxiliary components to the one or more detected auxiliary components of the 2D dentition image by comparing the one or more target locations of the expected auxiliary components with the one or more locations of the respective auxiliary components in the 2D dentition image. Based on such matching, processing logic may identify locations at which auxiliary components were expected but were not detected, which may be missing auxiliary components. In some embodiments, the auxiliary component detector may reject or filter out detected auxiliary components that fall outside expected locations to improve precision. For example, if the auxiliary component detector detects an auxiliary component at a location that does not correspondAttorney Docket No.: 28510.983 (L0820PCT)to any expected auxiliary component location from the 3D dentition model, the auxiliary component detector may determine that the detection is a false positive and may discard or ignore the detection. This filtering may significantly improve the precision of auxiliary component detection by eliminating spurious detections that do not align with the treatment plan. In another example, the one or more locations of the auxiliary components in the 2D dentition image may be projected onto the 3D dentition model. If no location of an auxiliary component is within a threshold distance from a target location, then the auxiliary component detector may determine that the expected auxiliary component corresponding to the target location is not present in the 2D dentition image.

[0073] As an example, the auxiliary component detector may determine a location for some auxiliary components disposed on a patients dentition, but may not determine a location for one auxiliary component (e.g., because the auxiliary component may have fallen out and the auxiliary component detection model does not detect the auxiliary component). The auxiliary component detector may project the locations of the expected auxiliary components of the 3D dentition model onto the 2D dentition image. The projected locations may be the target locations. The target locations of the expected auxiliary components may be compared to the locations of the auxiliary components. The auxiliary component detector may determine that the locations for some of the auxiliary components are within a threshold distance of the target locations for the expected auxiliary components. However, the auxiliary component detector may determine that there is no auxiliary component within a threshold distance of the target location corresponding to one of the expected auxiliary components. Thus, the auxiliary component detector may determine that the expected auxiliary component is not present in the 2D dentition image and is missing.

[0074] At block 150, processing logic may determine that at least one auxiliary component of the one or more expected auxiliary components is missing from the 2D dentition image. For example, as discussed above, the auxiliary component detector may determine that no location of an auxiliary component is within a threshold distance of the target location of one of the auxiliary components. In response, the auxiliary component detector may cause an indication of the missing auxiliary component from the 2D dentition image to be output on a III of a client computing device (e.g., a phone, a laptop / desktop, a tablet) of the patient or a dental professional associated with the patient. In some embodiments, before outputting the indication to a dental professional, the system may flag the 2D dentition image for human review and verification. For example, the system may add the 2D dentition image to a review queue where a human reviewer can verify whether the auxiliary component is actually missing before a notification is sent to the dental professional. This human verification step may reduce false alarms and improve the accuracy of notifications sent to dental professionals. In some embodiments, the III may include the III of the dental professional device 130. The auxiliary componentAttorney Docket No.: 28510.983 (L0820PCT)detector may cause the dental treatment computing system to send data to the dental professional device, and the data may include data identifying the patient, data identifying the tooth from which the auxiliary component 204A is missing (e.g., a tooth label), data identifying the component type of the missing auxiliary component, or other data. The data sent to the dental professional device may include an email, a push notification, or custom data compatible with software executing on the dental professional device. The III may present information indicated by the received data. In some embodiments, the auxiliary component detector may cause the dental treatment computing system to send similar data to the patient device, and a III of the patient device may present the information informing the patient of the missing auxiliary component. The dental professional can then contact the patient so the patient can have the dental professional replace the missing auxiliary component, or the patient may contact the dental professional to have the missing auxiliary component replaced.

[0075] In one implementation, the auxiliary component detector may generate a binary mask based on the registered 3D dentition model. The binary mask may include data indicating one or more locations in the 2D dentition image that correspond to the one or more expected auxiliary components. For example, the binary mask may include a matrix that includes cells corresponding to the respective pixels of the 2D dentition image. Each cell may include a value indicating whether the corresponding pixel is expected to contain an expected auxiliary component (e.g., “0” for does not contain, “1 ” for does contain). The auxiliary component detector may determine the value for a cell by projecting the portion of the registered 3D dentition model that corresponds to the pixel of the 2D dentition image that corresponds to the cell. If the portion of the registered 3D dentition model contains an auxiliary component, then the cell value may indicate an expected auxiliary component. If the portion of the registered 3D dentition model does not contain an auxiliary component, then the cell value may indicate an absence of an expected auxiliary component. The auxiliary component detector may then modify the 2D dentition image based on the binary mask. Modifying the 2D dentition image based on the binary mask may include adding a channel to each pixel of the 2D dentition image, and the value of the channel may be the value of the binary mask for the cell corresponding to the respective pixel. An Al model (e.g., the auxiliary component detection model) may then use the modified 2D dentition image as input, and the data provided by the binary data incorporated into the modified 2D dentition image may assist the auxiliary component detection model to detect auxiliary components 204 more accurately during inference.

[0076] In one embodiment, the auxiliary component detection model may output additional component information. The additional component information may include a tooth label corresponding to a respective location of an auxiliary component. The tooth label may include a label that uniquely identifies the tooth at the respective location (e.g., “UR1” for upper right 1, etc.). In some embodiments,Attorney Docket No.: 28510.983 (L0820PCT)the additional component information may further indicate a component type for the auxiliary component associated with the respective location. The component type may be an orthodontic button, a particular model or type of orthodontic button, an orthodontic attachment, a particular model or type of orthodontic attachment, or another type of auxiliary component.

[0077] FIG. 1B illustrates a flow diagram of an example method 180 for training a model for detecting auxiliary components, in accordance with some embodiments of the present disclosure. At block 182, processing logic obtains a 2D dentition image. The 2D dentition image may include a depiction of a patients dentition and one or more first auxiliary components depicted in the 2D dentition image. In one embodiment, a training data labeler may obtain the 2D dentition image, as discussed below in relation to FIG.4A.

[0078] At block 184, processing logic obtains a 3D dentition model. The 3D dentition model may include a depiction of the patient's dentition. The 3D dentition model may be obtained from a treatment plan associated with the patient's orthodontic treatment. The 3D dentition model may represent the patient's dentition ata particular stage of treatment, such as an initial stage, an intermediate stage, or a final target stage of the orthodontic treatment. The 3D dentition model may indicate one or more second auxiliary components depicted in the 3D dentition model, where the second auxiliary components correspond to expected auxiliary components that should be presenton the patient's teeth at the particular stage of treatment. In one embodiment, the training data labeler may obtain the 3D dentition model.

[0079] At block 186, processing logic registers the 3D dentition model to the 2D dentition image. For example, the training data labeler may use the auxiliary component detector to register the 3D dentition model to the 2D dentition image. Registering the 3D dentition model to the 2D dentition image may include aligning the 3D dentition model with the 2D dentition image, as discussed herein.

[0080] At block 188, processing logic corresponds the one or more second auxiliary components of the 3D dentition model with the one or more first auxiliary components of the 2D dentition image based at least in part on information from the 3D dentition model. For example, the auxiliary component detector may perform a classifying, segmenting, detection, recognition, etc. task to match auxiliary components depicted in the 3D dentition model correspond to corresponding auxiliary components depicted in the 2D dentition image. The training data labeler may identify the locations of the auxiliary components in the 2D dentition image based on the information from the 3D dentition model and label the locations.

[0081] At block 190, processing logic generates a training dataset item. The training dataset item can include a training inputthat includes the 2D dentition image. The training dataset item can include a label that includes the one or more locations of the auxiliary components of the 2D dentition image. TheAttorney Docket No.: 28510.983 (L0820PCT)locations may include the locations identified and labeled by the training data labeler. In one embodiment, the training data labeler may generate the training data item and add the training data item to the training dataset.

[0082] At block 192, processing logic trains an Al model on a training dataset. The Al model may be a model used to determine whether one or more expected auxiliary components are present in a 2D dentition image. The Al model may be the auxiliary component detection model discussed herein or another machine learning model discussed herein. T raining the Al model may include providing the 2D dentition images as training inputs and the associated labels specifying presence and / or location of auxiliary components (e.g., via segmentation information, bounding boxes, etc.) as target outputs. The training process may involve iteratively adjusting model parameters to minimize a loss function that measures the difference between predicted outputs and the target outputs. In some embodiments, the Al model may be trained as an object classification model that receives a 2D dentition image as input and outputs an indication of locations and / or presence of auxiliary components at one or more teeth. In some embodiments, the Al model may be trained with a "false alarm" or "false object" class in addition to classes for auxiliary components and background. The false alarm class may be used to train the model to distinguish between actual auxiliary components and objects that visually resemble auxiliary components but are not auxiliary components. For example, certain tooth features, reflections, or other visual artifacts may appear similar to attachments or buttons. By training the model with examples of such false alarms, the model may learn to more accurately identify actual auxiliary components and reduce false positive detections.

[0083] In some embodiments, the method 180 further includes obtaining tooth labels corresponding to the one or more locations of the one or more auxiliary components and including the tooth labels in the label of the training dataset item, as discussed above. The method 180 may further includes obtaining additional auxiliary component information (e.g., component type) and including the additional auxiliary component information in the label of the training dataset item. For example, the Al model may be trained to identify the specific teeth for which auxiliary components are detected and / or the specific types of auxiliary components that are detected at those teeth.

[0084] In one embodiment, the method 180 further includes detecting an incorrectly labeled item in the training dataset. The training data labeler may then remove the incorrectly labeled item or modify the item to correct the label. For example, the method 180 may include obtaining an item of training data from the training dataset. The item may include, as a training input, a 2D dentition image. The method 180 may include obtaining a 3D dentition model that corresponds to the 2D dentition image. The method 180 may include determining that one or more locations of the label of the training data item do not match the one or more target locations indicated by the 3D dentition model. For example,Attorney Docket No.: 28510.983 (L0820PCT)as discussed above, the auxiliary component detector may may determine that a location of an auxiliary component of the 3D dentition model is not contained within a bounding box of the 2D dentition image. In another example, the auxiliary component detector may segment the 2D dentition image by classifying regions the 2D dentition image into one or more dental classes (e.g., teeth, auxiliary components, etc.), and the training data labeler may determine that a location for an auxiliary component class does not correspond to a location of an auxiliary component indicated by the 3D dentition model.

[0085] The method 180 may further include, in response to the training data labeler determining that the training data item is incorrectly labeled, removing the item from the training dataset. In some embodiments, the method 180 further includes in response to the training data labeler determining that the training data item is incorrectly labeled, modifying the one or more locations of the label so that the locations correspond to the locations of the auxiliary components as indicated by the 3D dentition model.

[0086] FIG. 1C illustrates a flow diagram of an example method 130 for detecting auxiliary components using per-tooth processing, in accordance with some embodiments of the present disclosure. At block 132, processing logic obtains a 2D dentition image. The 2D dentition image may include a depiction of a patient's dentition. The 2D dentition image may be similar to the 2D dentition image 200 of FIG.2A or the 2D dentition image 260 of FIG.2C in embodiments. The 2D dentition image may include visual features corresponding to one or more auxiliary components such as orthodontic buttons, orthodontic attachments, power ridges, or orthodontic brackets. Obtaining the 2D dentition image may include an auxiliary component detector retrieving the 2D dentition image from dentition images of a dental treatment datastore and / or receiving the dentition image from a patient device. The patient device may have used a camera associated with the patient device to capture the 2D dentition image and may have provided the 2D dentition image to a dental treatment computing system.

[0087] At block 134, processing logic processes the 2D dentition image using one or more first models. The one or more first models output information identifying locations of a plurality of teeth in the 2D dentition image. In one embodiment, the one or more first models may include an artificial intelligence (Al) model trained to perform object detection to detect teeth, wherein the Al model outputs bounding boxes around each of the plurality of teeth in the 2D dentition image. The bounding boxes may define rectangular regions that encompass individual teeth within the image. The rectangular regions for teeth may also include depictions of portions of other adjacent teeth in embodiments. In some embodiments, the Al model identifies each tooth (e.g., identifies a tooth as a particular molar, canine, etc.). Teeth may be identified according to various tooth numbering standards used in dentalAttorney Docket No.: 28510.983 (L0820PCT)practice. In some embodiments, teeth may be identified using the Universal Numbering System, which assigns numbers 1 through 32 to permanent teeth in adults, starting from the upper right third molar (tooth 1) and proceeding around the upper arch to the upper left third molar (tooth 16), then continuing from the lower left third molar (tooth 17) around the lower arch to the lower right third molar (tooth 32). In some embodiments, teeth may be identified using the Palmer Notation Method, which divides the mouth into four quadrants and uses numbers 1 through 8 for each quadrant, with symbols indicating the specific quadrant. In some embodiments, teeth may be identified using the FDI World Dental Federation notation (also known as ISO 3950), which uses a two-digit numbering system where the first digit indicates the quadrant (1 -4 for permanent teeth, 5-8 for primary teeth) and the second digit indicates the tooth position within that quadrant (1-8). For example, in the FDI notation, the upper right central incisor may be designated as UR1 or tooth 11, where the first digit indicates the upper right quadrant and the second digit indicates the central incisor position. The tooth numbering information may be associated with auxiliary component locations to enable accurate identification of which teeth have attached auxiliary components and which teeth are missing expected auxiliary components. Although embodiments are described herein with reference to detection of auxiliary components, the per-tooth processing approach may also be applied to detection of other dental features of interest. For example, the systems and methods may be used to detect dental caries (cavities) or gingival recession on a per-tooth basis. In such embodiments, the one or more second models may be trained to identify dental caries or gingival recession rather than or in addition to auxiliary components, and the output may indicate which teeth have dental caries or gingival recession detected.

[0088] In embodiments, the one or more first models may include an Al model trained to perform segmentation of teeth in images, wherein the Al model outputs segmentation information for the plurality of teeth in the 2D dentition image. The segmentation information may include pixel-level classifications indicating which pixels belong to which teeth. In some embodiments, both object detection and segmentation may be performed, with the segmentation information providing more precise tooth boundaries than bounding boxes alone. The segmented dentition image 270 of FIG.2E illustrates an example of segmentation output showing individual tooth segments for both upper and lower dental arches.

[0089] At block 135, processing logic may perform one or more operations to identify locations of expected auxiliary components. This may include performing the operations of method 150 of FIG. 1 D and / or of method 160 of FIG. 1E in some embodiments. In some embodiments, this operation may be optional. In embodiments where this operation is performed, processing logic may obtain a 3D dentition model comprising a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components. Processing logic may register the 3D dentition model to the 2DAttorney Docket No.: 28510.983 (L0820PCT)dentition image and determine expected locations of the one or more expected auxiliary components based at least in part on information from the 3D dentition model, such as by using the techniques described with reference to FIGS. 1 A-B. The expected locations may be used to generate cropped 2D images that are focused on specific locations where auxiliary components are expected rather than on entire teeth. In some embodiments, a treatment plan may indicate which teeth should have auxiliary components attached thereto. Once the teeth in the 2D dentition image have been identified, processing logic may determine which of those teeth in the 2D dentition image are expected to have auxiliary components attached thereto based on the information provided in the treatment plan. This may be done without registering any image data of associated with the treatment plan to the 2D dentition image in embodiments. In some embodiments, processing logic may determine expected locations of auxiliary components by projecting the auxiliary components from the 3D dentition model onto the plane of the 2D dentition image following registration, thereby identifying precise pixel locations in the 2D dentition image where each auxiliary component should appear.

[0090] In some embodiments, no treatment plan may be available, such as in cases involving patients with wires and brackets who are not undergoing aligner-based treatment. In such embodiments, processing logic may use a prior detection of auxiliary components as a ground truth for comparison with current detections. For example, processing logic may obtain a first 2D dentition image captured at an earlier time, such as at the beginning of treatment or at a previous monitoring session. Processing logic may process the first 2D dentition image to detect auxiliary components present in the first 2D dentition image and establish the detected auxiliary components as the ground truth representing the expected state of the patient's dentition. Processing logic may then obtain a second 2D dentition image captured at a later time and process the second 2D dentition image to detect auxiliary components present therein. Processing logic may compare the auxiliary components detected in the second 2D dentition image to the auxiliary components detected in the first 2D dentition image to identify any differences, such as auxiliary components that were present in the first 2D dentition image but are missing from the second 2D dentition image. This approach enables monitoring of patients for missing auxiliary components even when treatment plan information is not available, by treating the initial detected state as the baseline against which subsequent images are compared. Processing logic may output a notice of any differences detected between the prior and current auxiliary component detections, alerting a dental professional or the patient to potential issues such as debonded brackets or missing buttons.

[0091] At block 136, processing logic generates one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth in the 2D dentition image. Each cropped 2D image of the one or more cropped 2D images is associated with aAttorney Docket No.: 28510.983 (L0820PCT)tooth of the plurality of teeth and comprises pixels of the 2D dentition image for the tooth and excludes pixels of the 2D dentition image for a remainder of the plurality of teeth. In many cases, the cropped images will still include some pixels associated with adjacent teeth to the tooth associated with a cropped 2D dentition image. The cropped 2D images may be generated by extracting rectangular regions from the 2D dentition image based on bounding boxes determined from the object detection or segmentation operations. In some embodiments, a cropped 2D image may be associated with a tooth and an expected auxiliary component on the tooth, and the cropped 2D image may comprise pixels of the 2D dentition image for the associated expected auxiliary component and exclude at least some pixels of the 2D dentition image for the tooth. For example, the cropped 2D image may be cropped to show the region where the auxiliary component should be, and may not show all of the tooth.Alternatively, the cropped 2D image may show all of the tooth. The cropped 2D image may have a size that is larger than a size of the expected auxiliary component in the 2D dentition image by up to a threshold number of pixels in some embodiments. In embodiments where a the 2D dentition image is cropped around individual auxiliary component locations rather than around individual teeth, multiple cropped 2D images may be generated for a single tooth, where each of the cropped 2D images is associated with a different auxiliary component expected for that tooth. For example, a treatment plan may indicate that a particular tooth should have both an attachment and a button, and a separate cropped image may be generated for each of the expected location of the attachment and the expected location of the button. Accordingly, in some embodiments, the one or more cropped 2D images may include a first cropped 2D image associated with a tooth and a first expected auxiliary component on the tooth and a second cropped 2D image associated with the tooth and a second expected auxiliary component on the tooth. The cropped 2D dentition image 280 of FIG. 2F is an example cropped image focused on an individual tooth. The cropped 2D dentition image 292 of FIG.2G is an example cropped image focused on an individual auxiliary component on a tooth.

[0092] In some embodiments, at block 138 processing logic identifies, for a cropped 2D image of the one or more cropped 2D images and based on the segmentation information, one or more pixels of the cropped 2D image that do not belong to the tooth associated with the cropped 2D image. The segmentation information may indicate which pixels within the cropped region belong to the tooth of interest and which pixels belong to adjacent teeth, gums, or other oral structures. This identification allows for isolation of the tooth of interest from surrounding structures that may otherwise interfere with auxiliary component detection.

[0093] In some embodiments, at block 140 processing logic modifies the cropped 2D image by setting values for the one or more pixels not associated with the tooth of interestto a predetermined value, wherein the one or more pixels having the predetermined value are ignored by the one or moreAttorney Docket No.: 28510.983 (L0820PCT)second models. In one embodiment, the predetermined value is a value for black (e.g., RGB values of [0, 0, 0]). By setting pixels that do not belong to the tooth of interest to black, the modified cropped 2D image focuses the attention of subsequent classification models on the tooth of interest while eliminating potential confusion from adjacent teeth or other structures. FIG.2H illustrates a cropped 2D image 286 that corresponds to cropped 2D image 280 of FIG. 2F, but with a masked region 290 masking off pixels not associated with a tooth of interest 288. The masked region 290 of FIG. 2H illustrates an example of pixels that have been set to a predetermined value (e.g., black) to mask out areas not belonging to the tooth of interest 288.

[0094] In some embodiments, at block 141 processing logic performs one or more operations to determine second model(s) to use to process the cropped 2D image. In some embodiments, the one or more second models may comprise a plurality of classification models, each trained to perform classification for a different type of auxiliary component. For example, the plurality of classification models may include a first model that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and a fourth model that identifies brackets. In some embodiments, rather than having a single model with multiple outputs (e.g., seven outputs for different combinations of orthodontic objects), the system may employ multiple separate models, where each model is specialized for detecting a particular type of auxiliary component. This approach may provide increased accuracy and reduce the likelihood of false positives for certain auxiliary component types. Processing logic may determine, based on a treatment plan, one or more types of expected auxiliary components for a tooth of the plurality of teeth. The treatment plan information may indicate which teeth are expected to have which types of auxiliary components (e.g., attachments, buttons, brackets). Processing logic may then determine, for a cropped 2D image of the one or more cropped 2D images that is associated with the tooth, which classification models of the plurality of classification models to apply to the cropped 2D image based on the one or more types of expected auxiliary components. For example, if the treatment plan indicates that a particular tooth is supposed to have a button, processing logic may run the button classifier on the cropped 2D image for that tooth to determine whether the button is present (e.g., zero or one buttons). Similarly, if the treatment plan indicates that a tooth has an attachment, processing logic may run the attachment classifier to determine whether the attachment is present (e.g., zero, one, or two attachments). The cropped 2D image may be processed using the determined classification models and may not be processed by a remainder of the plurality of classification models. This selective application of models may improve efficiency and reduce false positives by only running models that are relevant to the expected auxiliary components for each tooth. In some embodiments, when treatment plan information is not available (e.g., for wires and brackets cases where no treatment plan exists), processing logic may run all of the plurality of classificationAttorney Docket No.: 28510.983 (L0820PCT)models on each cropped 2D image and assess the outputs to determine which auxiliary components are present. In such cases, the initial set of photos may be used to establish a ground state representing the current configuration of auxiliary components, and subsequent photos may be compared against this ground state to detect any changes or missing components. In some embodiments, one or a few Al models (e.g., classification models) are trained to identify multiple different types of auxiliary components. In such embodiments, the operations of block 141 may be omitted.

[0095] At block 142, processing logic processes the one or more cropped 2D images using one or more second models. For each processed cropped 2D image, the one or more second models output an indication of whether the tooth associated with the cropped 2D image has one or more attached auxiliary components. The one or more second models may comprise one or more classification models trained to perform image classification. In some embodiments, the one or more classification models may include a single model that classifies multiple types of auxiliary components. In other embodiments, the one or more classification models may include a plurality of classification models, each trained to perform classification for a different type of auxiliary component, such as a first model that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and a fourth model that identifies brackets. The output of the classification models may include confidence values indicating the likelihood that each type of auxiliary component is present on the tooth.

[0096] In situations where multiple different types of auxiliary components are identified for a tooth, disambiguation logic may be applied to resolve which auxiliary components are actually present. The disambiguation logic may be rule-based, probabilistic, or a combination of both. Rule-based disambiguation may involve determining that certain types of auxiliary components cannot coexist on the same tooth. For example, a rule may specify that brackets and attachments cannot coexist on the same tooth, as having both would be clinically impractical. When the classification models indicate that both a bracket and an attachment are presenton a tooth, the disambiguation logic may select the type of auxiliary component having the highest confidence value. Probabilistic disambiguation may involve applying different confidence thresholds based on the rarity of certain auxiliary component types. For instance, if a button and an attachment are both detected on a tooth, and buttons are considered rare events, the confidence threshold for confirming the presence of a button may be set higher (e.g., 0.9) than for an attachment (e.g., 0.5). In some cases, multiple types of auxiliary components may legitimately coexist on the same tooth, such as a button and an attachment. When the confidence values for multiple auxiliary component types satisfy the applicable criteria and the auxiliary componentAttorney Docket No.: 28510.983 (L0820PCT)types can coexist according to the rules, the system may determine that multiple auxiliary component types are present at the tooth.

[0097] The embodiments described herein may utilize image classification rather than object detection for processing the cropped 2D images. Object detection and image classification are distinct machine learning tasks with different characteristics. Object detection involves identifying and localizing one or multiple objects within an image by outputting bounding boxes around detected objects along with their classifications. In contrast, image classification involves assigning one or more class labels to an entire image without providing location information for objects within the image. When performing per-tooth processing using cropped 2D images, image classification offers several advantages over object detection. First, image classifiers are generally simpler to train and deploy than object detection models, as they do not require bounding box annotations in the training data. Second, because the cropped 2D image is already focused on a single tooth, there is no need to localize auxiliary components within the image— the classification task is simply to determine whether an auxiliary component of a given type is presenter absent. Third, image classifiers may exist in greater variety than object detection models, allowing more flexibility in model selection and deployment. Fourth, the assignment of auxiliary components to individual teeth is inherently resolved by the per-tooth cropping approach, eliminating the need fora separate assignment step that may introduce errors when auxiliary components are positioned near tooth boundaries. By performing image classification on cropped per-tooth images rather than object detection on full dentition images, the system can achieve improved accuracy while simplifying the overall detection pipeline. However, in some embodiments, object detection may be performed on cropped per-tooth images rather than classification. Using object detection on cropped per-tooth images may enable detection of multiple auxiliary components within a single tooth along with their respective locations within the cropped image. For example, if a tooth has both an attachment and a button, an object detection model may output bounding boxes for both the attachment and the button within the cropped per-tooth image, along with classifications for each detected object. This approach may be advantageous when location information for auxiliary components within a tooth is desired, or when multiple auxiliary components of the same or different types may be presenton a single tooth.

[0098] At block 144, processing logic outputs an indication on a user interface of which teeth have attached auxiliary components. The indication may be output to a user interface of a patient device or a dental professional device. The indication may identify specific teeth that have auxiliary components attached, specific teeth that are missing expected auxiliary components, or both. In some embodiments, the indication may include information about the type of auxiliary component detected or missing for each tooth.Attorney Docket No.: 28510.983 (L0820PCT)

[0099] In some embodiments, the operations of blocks 134-142 of method 130 may be performed by a server computing device that interfaces with a client device over a network. The client device may obtain the 2D dentition image and transmit the 2D dentition image to the server computing device via the network. The server computing device may then process the 2D dentition image using the one or more first models to identify locations of teeth, generate the one or more cropped 2D images, identify pixels that do not belong to the tooth associated with each cropped 2D image, modify the cropped 2D images by setting values for non-tooth pixels to a predetermined value, determine which second models to use, and process the cropped 2D images using the one or more second models to generate indications of whether teeth have attached auxiliary components. The server computing device may generate an output identifying which auxiliary components are present, which auxiliary components are missing, or both. The server computing device may transmit the output to the client device over the network, and the client device may output the results to a display.

[0100] In some embodiments, the server computing device may operate within a cloud computing environment. The cloud computing environment may include one or more virtual machines, containers, or serverless computing instances that execute the processing operations. The cloud computing environment may provide scalable computing resources that can be allocated based on demand, such as when multiple patients submit 2D dentition images for processing concurrently. The cloud computing environment may store the trained machine learning models and may load the models into memory when processing requests from client devices.

[0101] The client device may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, or another computing device. The client device may include a camera that captures the 2D dentition image. For example, a patient may use a mobile phone camera to capture one or more photographs of the patient's dentition and may transmit the photographs to the server computing device for processing. In some embodiments, the client device may include software such as a mobile application or web browser that provides a user interface for capturing images, transmitting images to the server, receiving results from the server, and displaying the results to the user.

[0102] In some embodiments, all of the operations of method 130 may be performed on the client device without transmitting data to a server computing device. The client device may store the trained machine learning models locally and may execute the processing operations using a processor of the client device. This approach may reduce latency by eliminating network communication and may enable processing when the client device does not have network connectivity. The client device may include a graphics processing unitor neural processing unit that accelerates execution of the machine learning models.Attorney Docket No.: 28510.983 (L0820PCT)

[0103] In some embodiments, a report of the results of method 130 may be transmitted to a dental professional. The report may identify which teeth have attached auxiliary components, which teeth are missing expected auxiliary components, confidence values associated with the determinations, or other information. The report may be transmitted from the server computing device to a computing device associated with the dental professional, such as a desktop computer or tablet at a dental office. Alternatively, the report may be transmitted from the client device to the dental professional's computing device. The dental professional may review the report and may contact the patient to schedule an appointment if one or more auxiliary components are determined to be missing. In some embodiments, the report may be stored in a dental treatment datastore and may be accessed by the dental professional through a web-based interface or dedicated application.

[0104] FIG. 1D illustrates a flow diagram of an example method 150 for detecting auxiliary components using a 3D dentition model, in accordance with some embodiments of the present disclosure. The method 150 may be performed in conjunction with the method 130 of FIG. 1 C, for example, as part of block 135 or as additional operations performed before, during, or after the method 130.

[0105] At block 152, processing logic obtains a 3D dentition model comprising a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components. The 3D dentition model may be similar to the 3D dentition model 300 of FIG. 3. The 3D dentition model may be associated with a current stage of dental treatment and may reflect an expected condition of the patient's teeth predicted for the current stage. The expected auxiliary components may include orthodontic attachments, buttons, power ridges, brackets, and / or other auxiliary components that are called for by a dental treatment plan.

[0106] At block 154, processing logic registers the 3D dentition model to the 2D dentition image. Registering the 3D dentition model to the 2D dentition image may include aligning the 3D dentition model with the 2D dentition image through feature extraction, matching, and transformation operations. The registration process may involve identifying corresponding features between the 3D dentition model and the 2D dentition image, generating a transformation matrix, and applying rotation, translation, and / or scaling operations to align the coordinate systems. In some embodiments, the registration may be performed using rigid registration, non-rigid registration, or a combination thereof. Rigid registration involves applying global transformations that preserve the shape and size of the 3D dentition model, including rotation, translation, and uniform scaling operations. Rigid registration assumes that the spatial relationships between teeth in the 3D dentition model correspond directly to those in the 2D dentition image and may be suitable when the 3D dentition model accurately represents the current state of the patient's dentition. Non-rigid registration, also referred to as deformableAttorney Docket No.: 28510.983 (L0820PCT)registration, allows for local deformations and warping of the 3D dentition model to better align with the 2D dentition image. Non-rigid registration may account for differences between the 3D dentition model and the actual patient dentition captured in the 2D dentition image, such as tooth movements that have occurred during treatment, variations in jaw positioning, or differences between the treatment stage represented by the 3D dentition model and the current state of the patient's teeth. Non-rigid registration techniques may include thin-plate spline transformations, free-form deformations using B-splines, or other deformable transformation models that enable localized adjustments to individual teeth or regions of the dentition. In some embodiments, a coarse-to-fine registration approach may be employed, where rigid registration is first performed to achieve an initial global alignment, followed by non-rigid registration to refine the alignment and account for local variations. The 3D dentition model and the 2D dentition image may each be segmented prior to registration to improve registration accuracy. For the 2D dentition image, a segmentation model may be applied to identify and delineate individual teeth within the image, generating a segmentation mask that assigns each pixel to a corresponding tooth or background region. Similarly, the 3D dentition model may include segmentation information that identifies individual tooth meshes or surfaces within the model. The segmented image data from the 2D dentition image may then be registered to the segmented data from the 3D dentition model, enabling more precise alignment by matching corresponding tooth segments between the two representations. This segmentation-based registration approach may reduce errors caused by variations in image quality, lighting conditions, or partial occlusion of teeth, as the registration can leverage the identified tooth boundaries and shapes rather than relying solely on raw image features. The segmented registration may also facilitate per-tooth correspondence, enabling the system to accurately map expected auxiliary component locations from the 3D dentition model to specific regions in the 2D dentition image. When non-rigid registration is employed with segmented data, the registration may be performed on a per-tooth basis, allowing individual teeth in the 3D dentition model to be independently transformed to align with their corresponding segments in the 2D dentition image, thereby accommodating variations in tooth positions that may have occurred since the 3D dentition model was generated.

[0107] At block 158, processing logic determines expected locations of the one or more expected auxiliary components based at least in part on information from the 3D dentition model. The expected locations may be determined by projecting the locations of auxiliary components from the 3D dentition model onto the 2D dentition image using the registration transformation. These expected locations may be used to generate cropped 2D images that are focused on specific locations where auxiliary components are expected, enabling per-auxiliary-component classification rather than per-tooth classification in some embodiments.Attorney Docket No.: 28510.983 (L0820PCT)

[0108] FIG. 1E illustrates a flow diagram of an example method 160 for selecting classification models based on treatment plan information, in accordance with some embodiments of the present disclosure. The method 160 may be performed in conjunction with the method 130 of FIG. 1 C, for example, as part of block 141.

[0109] At block 162, processing logic determines, based on a treatment plan, one or more types of expected auxiliary components for a tooth of the plurality of teeth. The tooth may be a tooth associated with a cropped 2D dentition image that depicts that tooth. The treatment plan may specify which types of auxiliary components are to be attached to each tooth at each stage of treatment. For example, the treatment plan may indicate that a particular tooth should have an attachment but not a button, while another tooth should have both an attachment and a button.

[0110] At block 164, processing logic determines, for a cropped 2D image of the one or more cropped 2D images that is associated with the tooth, which classification models of the plurality of classification models to apply to the cropped 2D image based on the one or more types of expected auxiliary components. The cropped 2D image is processed using the determined classification models and is not processed by a remainder of the plurality of classification models. For example, if the treatment plan indicates that a tooth should have an attachment but not a button, processing logic may apply an attachment classification model to the cropped 2D image but may not apply a button classification model. This selective application of models may reduce computational overhead and may reduce false positives by avoiding application of models for auxiliary component types that are not expected on the tooth.

[0111] FIG. 1F illustrates a flow diagram of an example method 166 for determining types of auxiliary components present at a tooth using multiple classification models, in accordance with some embodiments of the present disclosure. The method 166 may be performed in conjunction with the method 130 of FIG. 1 C, for example, as part of block 142 or as additional operations performed after block 142.

[0112] At block 168, processing logic processes cropped 2D image(s) using a plurality of classification models, each trained to perform classification for a different type of auxiliary component. An output of the plurality of classification models with reference to a cropped 2D image of the plurality of cropped 2D images is a plurality of confidence values, each associated with a different type of auxiliary component. For example, the plurality of classification models may include a button classification model, an attachment classification model, a power ridge classification model, and / or a bracket classification model, and each model may output a confidence value indicating the likelihood that the respective type of auxiliary component is present in the cropped 2D image.Attorney Docket No.: 28510.983 (L0820PCT)

[0113] At block 170, processing logic determines confidence values associated with each of the plurality of types of auxiliary components. The confidence values may be numerical values between 0 and 1, with higher values indicating greater confidence that the respective type of auxiliary component is present.

[0114] At block 172, processing logic determines which types of auxiliary components have associated confidence values that satisfy one or more criteria. The one or more criteria may include a first criterion that a confidence value associated with a type of auxiliary component is greater than confidence values associated with a remainder of types of auxiliary components. The one or more criteria may additionally or alternatively include a second criterion that the confidence value associated with the type of auxiliary component exceeds a confidence threshold. The confidence threshold may be a predetermined value such as 0.5, 0.7, 0.9, or another suitable value.

[0115] At block 174, processing logic determines one or more types of auxiliary components present at the tooth associated with the cropped 2D image based on satisfaction of the one or more criteria by the confidence value(s) associated with the one or more types of auxiliary components. Block 174 may include multiple sub-blocks 176-182 in some embodiments.

[0116] At block 176, processing logic determines whether the criteria are satisfied for multiple auxiliary component types. If the criteria are not satisfied for multiple auxiliary component types, the method 166 proceeds to block 180. If the criteria are satisfied for multiple auxiliary component types, the method 166 proceeds to block 178.

[0117] At block 178, processing logic determines whether the auxiliary component types can coexist. Certain combinations of auxiliary components may be unlikely or impossible to coexist on the same tooth. For example, a bracket and an attachment may be unlikely to coexist on the same tooth because they serve similar functions and may physically interfere with each other. In contrast, an attachment and a button may be able to coexist on the same tooth. Processing logic may apply a rule that indicates whether the multiple types of auxiliary components can coexist on the tooth. If the auxiliary component types cannot coexist, the method 166 proceeds to block 180. If the auxiliary component types can coexist, the method 166 proceeds to block 182.

[0118] At block 180, processing logic selects a single auxiliary component type. When multiple auxiliary component types have confidence values that satisfy the criteria but the auxiliary component types cannot coexist, processing logic may select the type of auxiliary component that satisfies a selection rule. The selection rule may be a rule to select a type of auxiliary component associated with a highest confidence value. Alternatively, the selection rule may incorporate prior probabilities or other factors to select the most likely auxiliary component type. In some embodiments, the selection rule may utilize contextual information from other teeth in the dentition to inform the selection. For example, if theAttorney Docket No.: 28510.983 (L0820PCT)classification models have detected attachments on multiple other teeth in the patient's dentition but have not detected any brackets, this contextual information may be used to determine that the current tooth likely includes an attachment rather than a bracket. Conversely, if the classification models have detected brackets on multiple other teeth but have not detected any attachments, this contextual information may be used to determine that the current tooth likely includes a bracket rather than an attachment. This approach leverages the observation that a patient's orthodontic treatment typically involves a consistent type of auxiliary component across the dentition— patients undergoing aligner treatment generally have attachments on their teeth, while patients undergoing traditional orthodontic treatment generally have brackets on their teeth. Accordingly, when the classification models produce ambiguous results for a particular tooth (e.g., both the attachment classifier and the bracket classifier produce confidence values that satisfy the threshold criteria), processing logic may examine the auxiliary component types detected on other teeth in the same dentition image to resolve the ambiguity. If a majority of other teeth have been classified as having attachments, processing logic may select the attachment classification for the current tooth. If a majority of other teeth have been classified as having brackets, processing logic may selectthe bracket classification forthe current tooth. This contextual disambiguation approach can improve overall detection accuracy by ensuring consistency across the dentition and reducing false positives that may arise from visual similarities between different auxiliary component types.

[0119] At block 182, processing logic selects multiple auxiliary component types. When multiple auxiliary component types have confidence values that satisfy the criteria and the auxiliary component types can coexist, processing logic may determine that each of the multiple types of auxiliary components are present at the tooth.

[0120] FIG. 1G illustrates a flow diagram of an example method 184 for training a model for detecting auxiliary components using per-tooth processing, in accordance with some embodiments of the present disclosure. The method 184 may include one or more actions discussed below in relation to a model training workflow 405B of FIG.4B.

[0121] At block 186, processing logic obtains a 2D dentition image including a depiction of a patient's dentition. The 2D dentition image may be similar to the 2D dentition image 200 of FIG. 2A, the 2D dentition image 260 of FIG.2C, or the 2D dentition image 264 of FIG.2D. The 2D dentition image may include visual features corresponding to one or more auxiliary components disposed on the patient's teeth.

[0122] At block 187, processing logic processes the 2D dentition image using one or more first models, wherein the one or more first models output information identifying locations of a plurality of teeth in the 2D dentition image. The one or more first models may include an Al model trained toAttorney Docket No.: 28510.983 (L0820PCT)perform object detection to detect teeth, wherein the Al model outputs bounding boxes around each of the plurality of teeth in the 2D dentition image. Alternatively or additionally, the one or more first models may include an Al model trained to perform segmentation of teeth in images, wherein the Al model outputs segmentation information for the plurality of teeth in the 2D dentition image.

[0123] At block 188, processing logic generates one or more cropped 2D images from the 2D dentition image based on the information identifying the locations of the plurality of teeth in the 2D dentition image. Each cropped 2D image of the one or more cropped 2D images is associated with a tooth of the plurality of teeth and comprises pixels of the 2D dentition image for the tooth and excludes pixels of the 2D dentition image for a remainder of the plurality of teeth. In some embodiments, rather than generating cropped 2D images based on individual teeth, processing logic may generate cropped 2D images based on expected locations of individual auxiliary components. In such embodiments, processing logic may first perform registration of a 3D dentition model to the 2D dentition image to determine expected locations of auxiliary components, and then generate cropped 2D images that are centered on each expected auxiliary component location rather than on each tooth. Each such cropped 2D image may be associated with a tooth and an expected auxiliary component on that tooth, and may comprise pixels of the 2D dentition image for the associated expected auxiliary component while excluding at least some pixels of the 2D dentition image for the tooth. The cropped 2D image may have a size that is larger than a size of the expected auxiliary component in the 2D dentition image by up to a threshold number of pixels to provide context around the auxiliary component. In embodiments where a tooth has multiple expected auxiliary components (e.g., both an attachment and a button), the one or more cropped 2D images may comprise a first cropped 2D image associated with the tooth and a first expected auxiliary component on the tooth and a second cropped 2D image associated with the tooth and a second expected auxiliary component on the tooth. This per-auxiliary-component approach may provide higher resolution images focused specifically on the regions where auxiliary components are expected, potentially improving detection accuracy compared to per-tooth cropping.

[0124] At block 190, processing logic identifies, for a cropped 2D image of the one or more cropped 2D images and based on the segmentation information, one or more pixels of the cropped 2D image that do not belong to the tooth associated with the cropped 2D image. In embodiments where the cropped 2D image is associated with an expected auxiliary component rather than an entire tooth, processing logic may identify one or more pixels of the cropped 2D image that do not belong to the expected auxiliary component or the region immediately surrounding the expected auxiliary component. In such embodiments, the cropped 2D image may be generated based on expected locations of auxiliary components determined from a registered 3D dentition model, and the cropped 2D image may have a size that is larger than a size of the expected auxiliary component in the 2D dentition image byAttorney Docket No.: 28510.983 (L0820PCT)up to a threshold numberof pixels, thereby providing sufficient context for classification while focusing on the specific location where the auxiliary component should be present.

[0125] In some embodiments, at block 192, processing logic modifies the cropped 2D image by setting values for the one or more pixels to a predetermined value, wherein the one or more pixels having the predetermined value are ignored by the one or more second models. The pixels that are set to the predetermined value include those pixels identified as not belonging to the tooth or auxiliary component associated with the cropped 2D image based on the segmentation information. For example, pixels corresponding to adjacent teeth, gum tissue, or other oral structures that appear within the bounding box of the cropped 2D image but are not part of the tooth of interest may be set to the predetermined value. In some embodiments, the predetermined value is a value for black (e.g., RGB value [0, 0, 0]), which effectively masks out the non-tooth regions and allows the one or more second models to focus on the tooth and any auxiliary components attached thereto without interference from surrounding structures. By blacking out pixels from neighboring teeth and other oral anatomy, the cropped 2D image presents a cleaner input to the classification models, potentially reducing confounding visual information and improving the accuracy of auxiliary component detection.

[0126] At block 194, processing logic determines, for each cropped 2D image of the one or more cropped 2D images, a label specifying whether the cropped 2D image includes an auxiliary component. The label may be determined through one or more of the operations at blocks 195 and 196.

[0127] At block 195, processing logic may obtain a 3D dentition model, register the 3D dentition model to the 2D dentition image, and determine whether auxiliary components are present for individual teeth in the 2D dentition image based on the 3D dentition model. The 3D dentition model may indicate one or more expected auxiliary components, and the registration may enable processing logic to determine which teeth in the 2D dentition image have auxiliary components based on the locations of auxiliary components in the 3D dentition model.

[0128] At block 196, processing logic may receive a manually entered label. A human labeler may review the cropped 2D image and provide a label indicating whether the cropped 2D image includes an auxiliary component and, optionally, the type of auxiliary component present.

[0129] At block 198, processing logic trains one or more second models to process an input of a cropped 2D image of a tooth and to output an indication of whether the tooth in the cropped 2D image has one or more attached auxiliary components based on the one or more cropped 2D images and associated labels. The one or more second models may comprise one or more classification models trained to perform image classification in some embodiments. In some embodiments, the one or more classification models may include a single model that classifies multiple types of auxiliary components. In other embodiments, the one or more classification models may include a plurality of classificationAttorney Docket No.: 28510.983 (L0820PCT)models, each trained to perform classification for a different type of auxiliary component, such as a first model that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and a fourth model that identifies brackets.

[0130] FIG.2A illustrates a 2D dentition image 200, according to certain embodiments. The 2D dentition image 200 may have been captured, for example, by a patient device or by a dental professional device that includes a camera. As shown, the 2D dentition image 200 includes one or more teeth 202A-I. The 2D dentition image 200 may include one or more auxiliary components 204A-C, F-G, I. An auxiliary component 204 may be attached to a tooth 202. The 2D dentition image 200 may further depict oral features surrounding the patients dentition, such as gums 206 and a tongue 208. An auxiliary component 204 may include an orthodontic button. An orthodontic button may include a structure disposed on a tooth 202 that is shaped and positioned to receive a force from an aligner to guide the movement of the tooth 202. An auxiliary component 204 may include an orthodontic attachment. An orthodontic attachment may include a structure disposed on a tooth 202 configured to attach to orthodontic hardware and to enable the orthodontic aligner to apply forces to the patients teeth.

[0131] The 2D dentition image 200 may include an image captured by the camera of a patient device. For example, where the patient device is a mobile phone, the patient may point the camera at their mouth and cause the camera to capture an image of the mouth. The patient may capture multiple 2D dentition images 200. Different 2D dentition images 200 may depict the same patients dentition from different angles or with the patients mouth in different positions (e.g., closed, partially open, completely open, etc.), which may show different teeth 202 or auxiliary components 204.

[0132] As shown, dental auxiliary components 204A-G may be transparent or colored to blend in with the teeth 202A-I. This can make detection of the auxiliary components 204A-G very difficult.Accordingly, embodiments described herein cover improved techniques for detecting such dental auxiliary components 204A-G using one or more Al models.

[0133] FIG.2B illustrates a 2D dentition image 250, according to certain embodiments. The 2D dentition image 250 may include the 2D dentition image 200 of FIG.2A with additional data. As seen in FIG.2B, the 2D dentition image 250 may include the one or more teeth 202A-I, the one or more auxiliary components 204A-C, F-G, I, the gums 206, and the tongue 208 of FIG. 2A. The 2D dentition image 250 may further include bounding boxes 252B-D, G. The bounding boxes 252 may surround auxiliary components 202B-D, G. The auxiliary component detector 112 may generate the bounding boxes 252 in response to identifying the auxiliary components 202B-D, G as being present in the 2D dentition image 200, as discussed herein. Where the 2D dentition image 250 is a labeled item of a training dataset for an auxiliary component-detecting Al model, the bounding boxes 252 may indicateAttorney Docket No.: 28510.983 (L0820PCT)that a human or software labeler identified the area surrounded by the bounding boxes as containing auxiliary components 204. An auxiliary component detector of a dental treatment computing system may use the bounding boxes 252 and information from the 3D dentition model to determine whether one or more auxiliary components 204 are present in the 2D dentition image 250 and / or determine whether one or more auxiliary components 204 are missing from the 2D dentition image 250.

[0134] As seen in FIG.2B, the auxiliary components 204A and 204I do not have corresponding bounding boxes 252. This may be because the auxiliary component detector 112 or a human or software labeler may have failed to identify the auxiliary components 204A and 204I. If the bounding boxes 252 were produced by a human or software labeler, the dental treatment computing system may flag the 2D dentition image 250 for human review follow up, and bounding boxes 252 for auxiliary components 204A and 204I can be added. If the bounding boxes 252 were produced by the auxiliary component detector 112, the auxiliary component detector may (incorrectly) determine that the auxiliary components 204A and 204I are not present in the 2D dentition image 250 and / or determine that the auxiliary components 204A and 204I are missing from the 2D dentition image 250. Further review of the dentition image 250 may indicate that the auxiliary component detector 112 generated an incorrect output and that further refinement or training is needed.

[0135] FIG.2C illustrates a 2D dentition image 260, according to certain embodiments. The 2D dentition image 260 depicts a frontal view of a patient's dentition showing both the upper and lower dental arches in a closed bite position. The image captures multiple teeth across both dental arches, with the upper teeth visible above and the lower teeth visible below. The gingival tissue surrounding the teeth is visible in both the upper and lower regions of the image. The 2D dentition image 260 may be captured by a camera of a patient device or a dental professional device. In some embodiments, the 2D dentition image 260 may be captured while the patient is wearing a lip retractor, cheek retractor, or similar oral retraction device that holds the lips and cheeks away from the teeth to provide an unobstructed view of the dentition. Such retraction devices facilitate improved visibility of the teeth and any auxiliary components attached thereto by preventing the lips and cheeks from obscuring portions of the dental arches. In other embodiments, the 2D dentition image 260 may be captured without the use of a retraction device, with the patient simply retracting their lips manually or smiling to expose the teeth. The 2D dentition image 260 may include visual features corresponding to one or more auxiliary components disposed on the teeth, which may be difficult to identify due to the auxiliary components being colored to match the color of the patient's teeth. The 2D dentition image 260 may be used as input for processing by an auxiliary component detector to determine whether expected auxiliary components are presenter missing from the patient's dentition.Attorney Docket No.: 28510.983 (L0820PCT)

[0136] FIG.2D illustrates a 2D dentition image 264, according to certain embodiments. The 2D dentition image 264 depicts a patient's upper and lower dental arches in a frontal view. The 2D dentition image 264 shows multiple teeth 268 of the patient's dentition with bounding boxes 266 positioned around various locations on the teeth 268. The bounding boxes 266 are represented as rectangular outlines that indicate detected or expected locations of auxiliary components on the teeth. Multiple bounding boxes 266 are distributed across both the upper and lower arches, with several appearing on the upper teeth near the gumline and additional bounding boxes 266 positioned on the lower teeth. The 2D dentition image 264 demonstrates how bounding boxes 266 can be used to identify and mark locations of auxiliary components within a 2D dentition image, with the auxiliary component 268 being specifically indicated for detection or classification purposes in the context of orthodontic object detection systems.

[0137] For certain images, issues may arise from training a machine learning model using the whole image. First, auxiliary components such as orthodontic attachments and buttons are often small and tooth colored, making their identification difficult. Reducing the resolution of the image to something consumable by a machine learning model may make this identification even more difficult, potentially leading to a reduction in accuracy. For example, the auxiliary component 268 highlighted in the 2D dentition image 264 may be an attachment that could be difficult to detect if the image resolution was decreased significantly. Second, assignmentof auxiliary components to individual teeth may be complicated because the auxiliary component may be very close to adjacent teeth. For example, the auxiliary component 268 may overlap significantly with an adjacent tooth. A machine learning model prediction of the bounding box for the auxiliary component 268, which is predicted with error, could potentially be assigned to the wrong tooth. These challenges may be addressed by the per-tooth processing approaches described herein, which enable higher resolution analysis of individual teeth and eliminate the need for separate assignmentof detected auxiliary components to teeth.

[0138] FIG.2E illustrates a segmented dentition image 270, according to certain embodiments. The segmented dentition image 270 shows a front view of a patient's upper and lower dental arches with individual teeth identified through segmentation. The segmented dentition image 270 displays the results of a tooth segmentation process (e.g., as performed on the 2D dentition image 260 of FIG.2C) where each tooth has been identified and assigned a distinct visual representation. Each tooth may also be identified according to a dental numbering system and labeled with such information. For example, each tooth segment may be assigned a particular tooth number according to a tooth numbering system. The upper dental arch includes multiple tooth segments 272A-272L arranged from left to right. The lower dental arch similarly includes multiple tooth segments 274A-274L. Each tooth segment in the segmented dentition image 270 is represented with a different color or otherAttorney Docket No.: 28510.983 (L0820PCT)visualization to visually distinguish individual teeth from one another. This segmentation information can be used to identify bounding boxes around individual teeth, enabling the generation of cropped 2D images for per-tooth analysis of auxiliary components such as attachments, buttons, or brackets.

[0139] FIG.2F illustrates an example cropped 2D dentition image 280 comprising a depiction of a single tooth from a patient's dentition, in accordance with some embodiments of the present disclosure. The cropped 2D dentition image 280 is generated from a larger 2D dentition image based on information identifying the location of the tooth, such as bounding box information or segmentation information output by one or more first models. The cropped 2D dentition image 280 includes pixels corresponding to the tooth and may exclude pixels corresponding to other teeth in the patient's dentition. By focusing on a single tooth, the cropped 2D dentition image 280 provides higher resolution detail compared to processing the entire dentition image, which can improve detection accuracy for auxiliary components such as attachments, buttons, or brackets that may be difficult to identify due to their small size or tooth-colored appearance. The cropped 2D dentition image 280 may be processed by one or more second models, such as classification models, to determine whether the tooth has one or more attached auxiliary components.

[0140] FIG.2G illustrates the example 2D dentition image of FIG.2F with bounding boxes 284 identifying regions of interest associated with auxiliary components, in accordance with some embodiments of the present disclosure. The cropped 2D dentition image shown in FIG.2G has a higher resolution than the original full dentition image when focused on a single tooth, leading to easier identification of auxiliary components. This cropped image, paired with an attachment label indicating the presence or absence of one or more auxiliary components on the depicted tooth, may be used as a data point in a training dataset used to train a machine learning (ML) classifier. The attachment label may specify the type of auxiliary component (e.g., attachment, button, bracket) and may be derived from human annotations or from registration of a 3D dentition model to the 2D dentition image. By generating multiple such cropped images from full dentition images and pairing each with corresponding labels, a training dataset can be constructed fortraining an image classifier to detect auxiliary components on a per-tooth basis, as opposed to training an object detection model on whole images.

[0141] FIG.2H illustrates an example cropped 2D image with masked regions, in accordance with some embodiments of the present disclosure. As shown in FIG.2G, the cropped 2D image of a tooth of interest may include not only auxiliary components (e.g., attachments) associated with the tooth of interest but also auxiliary components from neighboring teeth that appear within the cropped region. For example, when a bounding box is generated around a particular tooth based on segmentation information, portions of adjacent teeth and their associated attachments may be captured within theAttorney Docket No.: 28510.983 (L0820PCT)cropped image boundaries. To reduce potential confusion during classification or detection operations, pixels from neighboring teeth, as identified by a tooth segmentation mask, can be set to black (e.g., a pixel value of [0, 0, 0]) or another value, thereby eliminating large portions of attachments from other teeth and reducing confounding visual information. FIG.2H depicts the result of this masking operation, where the masked regions corresponding to neighboring teeth have been set to the predetermined black value, allowing the classification model to focus on the tooth of interest and its associated auxiliary components without interference from adjacent dental structures.

[0142] FIG.2I illustrates an example cropped 2D image 292 comprising a depiction of a single auxiliary component, in accordance with some embodiments of the present disclosure. The cropped 2D image 292 is generated based on the bounding box 284 identified in FIG. 2G, where the cropped 2D image 292 includes pixels corresponding to the auxiliary component and a small surrounding region while excluding the remainder of the tooth and other dental structures. In embodiments where a 3D dentition model is registered to the 2D dentition image, the expected location of each auxiliary component can be projected onto the 2D dentition image, enabling the generation of cropped 2D images 292 that are centered on individual expected auxiliary components rather than on entire teeth. This per-auxiliary-component cropping approach provides several advantages over per-tooth cropping. First, the cropped 2D image 292 can maintain higher effective resolution for the auxiliary component because the image chip is sized to the auxiliary component rather than to the larger tooth area, allowing classification models to analyze finer details of the auxiliary component. Second, when a tooth has multiple expected auxiliary components, separate cropped 2D images 292 can be generated for each expected auxiliary component location, enabling independent classification of whether each individual auxiliary component is present without requiring the classification model to simultaneously detect and distinguish multiple auxiliary components on the same tooth. Third, the cropped 2D image 292 reduces potential confusion from adjacent auxiliary components on neighboring teeth that might otherwise appear in a per-tooth cropped image. The cropped 2D image 292 may be processed by a classification model trained to output an indication of whether the expected auxiliary component is present at the location depicted in the cropped 2D image 292.

[0143] FIG.3 illustrates a 3D dentition model 300, according to some embodiments. As shown, the 3D dentition model 300 can include one or more teeth 302A-I. In embodiments, the 3D dentition model 300 is a mesh. In embodiments, the 3D dentition model 300 is included in a dental treatment plan (e.g., an orthodontic treatment plan), and represents the planned dentition for a patient at a particular stage in treatment. The 3D dentition model 300 may include one or more auxiliary components 304A-C, F-G, I. An auxiliary component 304 may be attached to or otherwise associated with a tooth 302 in the 3D dentition model 300, and may reflect dental auxiliary components that were,Attorney Docket No.: 28510.983 (L0820PCT)or should have been, attached to the respective teeth of the patient at the respective planned locations and / or orientations as called for in the dental treatment plan. The 3D dentition model 300 may further depict oral features surrounding the patients dentition, such as gums 306. As discussed above, the 3D dentition model 300 may include data representing a 3D model of a patients dentition. The 3D dentition model 300 may include a 3D model provided to the dental treatment computing system 510 from a computing device used by an orthodontic professional (e.g., the dental professional device 130), and may have been generated as part of development of a dental treatment plan. As depicted in FIG. 3, each tooth 302 may be labeled to uniquely identify the respective tooth 302. For example, the tooth 302D is labeled “UR1 ” (upper right 1 ), the tooth 302E is labeled “UL1 ” (upper left 1 ), and so on.

[0144] As discussed above, the 3D dentition model 300 may include a 3D model generated by one or more computing devices. In one example, the patient may visit a dental professional’s office, and the dental professional may use a combination of hardware and software to scan the patients dentition and generate the 3D dentition model 300 (e.g., using intraoral and / or exterior scanner devices that scan the patients dentition from multiple positions). The software of the dental professional may then provide the 3D dentition model 300 to the dental treatment computing system for storage in the dentition models of a dental treatment datastore. In another example, the patient may use the camera of the patients device to capture multiple images of the patients dentition, and software of the patient device or software of the dental treatment computing system may receive those images and generate the 3D dentition model 300 based on those images.

[0145] In some embodiments, the 3D dentition model 300 may be generated before the 2D dentition image 200. For example, the 3D dentition model 300 may be a 3D model of the patients dentition that was generated during an initial phase of the patients orthodontic treatment (e.g., during treatment planning), and the 2D dentition image 200 may be an image captured by the camera of the patient device during a later, intermediate phase of the treatment.

[0146] FIG.4A illustrates a model training workflow 405A and a model application workflow 417A for a dental treatment computing system capable of detecting missing dental auxiliary components, in accordance with an embodiment of the present disclosure. In embodiments, the model training workflow 405A may be performed ata server, and the trained models are provided to a dental treatment computing system (e.g., on the dental treatment computing system 510 of FIG. 5), which may perform the model application workflow 417A. The model training workflow 405A and the model application workflow 417A may be performed by processing logic executed by a processor of a computing device. One or more of these workflows 405A, 417A may be implemented, for example, by one or more machine learning modules implemented in dental treatment computing system 510 of FIG.5.Attorney Docket No.: 28510.983 (L0820PCT)

[0147] The model training workflow 405A trains one or more machine learning models (e.g., deep learning models) to perform one or more classifying, segmenting, detection, recognition, etc. tasks for 2D dentition images in embodiments. The model application workflow 417A is to apply the one or more trained machine learning models to perform the classifying, segmenting, detection, recognition, etc. tasks for 2D dentition images to identify missing dental auxiliary components, and to perform one or more actions in response to detecting missing dental auxiliary components.

[0148] One type of machine learning model that may be used to perform some or all of the above asks is an artificial neural network, such as a deep neural network. Artificial neural networks generally include a feature representation component with a classifier or regression layers that map features to a desired output space. A convolutional neural network (CNN), for example, hosts multiple layers of convolutional filters. Pooling is performed, and non-linearities may be addressed, at lower layers, on top of which a multi-layer perceptron is commonly appended, mapping top layer features extracted by the convolutional layers to decisions (e.g. classification outputs). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks may learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks include a hierarchy of layers, where the different layers learn different levels of representations that correspond to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and composite representation. In an image recognition application, for example, the raw input may be a matrix of pixels; the first representational layer may abstract the pixels and encode edges; the second layer may compose and encode arrangements of edges; the third layer may encode higher level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer may recognize a scanning role. Notably, a deep learning process can learn which features to optimally place in which level on its own. The "deep" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs may be that of the network and may be the number of hidden layers plus one. For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited.

[0149] In one embodiment, a U-net architecture is used for one or more machine learning model. A U-net is a type of deep neural network that combines an encoder and decoder together, with appropriate concatenations between them, to capture both local and global features. The encoder is a series of convolutional layers that increase the number of channels while reducing the height and widthAttorney Docket No.: 28510.983 (L0820PCT)when processing from inputs to outputs, while the decoder increases the height and width and reduces the number of channels. Layers from the encoderwith the same image height and width may be concatenated with outputs from the decoder. Any or all of the convolutional layers from encoder and decoder may use traditional or depth-wise separable convolutions.

[0150] In one embodiment, one or more machine learning model is a recurrent neural network (RNN). An RNN is a type of neural network that includes a memory to enable the neural network to capture temporal dependencies. An RNN is able to learn input-output mappings that depend on both a current input and past inputs. The RNN will address past and future scans and make predictions based on this continuous scanning information. RNNs may be trained using a training dataset to generate a fixed number of outputs (e.g., to classify time varying data such as video data as belonging to a fixed number of classes). One type of RNN that may be used is a long short term memory (LSTM) neural network.

[0151] A common architecture for such tasks is LSTM. Unfortunately, LSTM is notwell suited for images since it does not capture spatial information as well as convolutional networks do. For this purpose, one can utilize ConvLSTM - a variant of LSTM containing a convolution operation inside the LSTM cell. ConvLSTM is a variant of LSTM (Long Short-Term Memory) containing a convolution operation inside the LSTM cell. ConvLSTM replaces matrix multiplication with a convolution operation at each gate in the LSTM cell. By doing so, it captures underlying spatial features by convolution operations in multiple-dimensional data. The main difference between ConvLSTM and LSTM is the number of input dimensions. As LSTM input data is one-dimensional, it is not suitable for spatial sequence data such as video, satellite, radar image data set. ConvLSTM is designed for 3-D data as its input. In one embodiment, a CNN-LSTM machine learning model is used. A CNN-LSTM is an integration of a CNN (Convolutional layers) with an LSTM. First the CNN part of the model processes the data and a one-dimensional result feeds an LSTM model.

[0152] In one embodiment, a class of machine learning model called a MobileNet is used for one or more neural networks. A MobileNet is an efficient machine learning model based on a streamlined architecture that uses depth-wise separable convolutions to build lightweight deep neural networks. MobileNets may be convolutional neural networks (CNNs) that may perform convolutions in both the spatial and channel domains. A MobileNet may include a stack of separable convolution modules that are composed of depthwise convolution and pointwise convolution (conv 1x1). The separable convolution independently performs convolution in the spatial and channel domains. This factorization of convolution may significantly reduce computational cost from HWNK2M to HWNK2(depthwise) plus HWNM (conv 1x1), HWN(K2-HM) in total, where N denotes the number of input channels, K2denotes theAttorney Docket No.: 28510.983 (L0820PCT)size of convolutional kernel, M denotes the number of output channels, and HxW denotes the spatial size of the output feature map. This may reduce a bottleneck of computational cost to conv 1x1.

[0153] Training of a neural network may be achieved in a supervised learning manner, which involves feeding a training dataset consisting of labeled inputs through the network, observing its outputs, defining an error (by measuring the difference between the outputs and the label values), and using techniques such as deep gradient descent and backpropagation to tune the weights of the network across all its layers and nodes such that the error is minimized. In many applications, repeating this process across the many labeled inputs in the training dataset yields a network that can produce correct output when presented with inputs that are different than the ones present in the training dataset. In high-dimensional settings, such as large images, this generalization is achieved when a sufficiently large and diverse training dataset is made available.

[0154] For the model training workflow 405, a training dataset containing hundreds, thousands, tens of thousands, hundreds of thousands or more intraoral scans, images and / or 3D models should be used to form a training dataset. In embodiments, up to millions of cases of patient dentition that may have undergone a prosthodontic procedure and / or an orthodontic procedure may be available for forming a training dataset, where each case may include various labels of one or more types of useful information. Each case may include, for example, data showing a 3D model, intraoral scans, height maps, color images, NIRI images, etc. of one or more dental sites, data showing pixel-level segmentation of the data (e.g., 3D model, intraoral scans, height maps, color images, NIRI images, etc.) into various dental classes (e.g., tooth, restorative object, gingiva, moving tissue, upper palate, etc.), data showing one or more assigned classifications for the data, data showing bounding boxes (e.g., bounding box coordinates and / or sizes), and so on. This data may be processed to generate one or multiple training datasets 436 for training of one or more machine learning models.

[0155] In one embodiment, generating one or more training datasets 436 includes gathering 2D dentition images 410, optionally with labels. In some embodiments, generating one or more training datasets 436 further includes labeling the 2D dentition images and / or updating labeling of one or more 2D dentition images 410 based on 3D models 415 corresponding to the 2D images 410. For example, a 2D image 410 may correspond to a particular treatment stage for a particular patient, and a 3D model 415 for that treatment stage and patient may be determined and used to check labeling of the 2D image 410, and / or to modify a labeling of and / or add labeling to the 2D image 410. The added / modified / checked labels may be with respect to dental auxiliary components on teeth of the dental arch in embodiments. The labels may indicate, for each 2D image 410, the positions, locations, shapes, etc. of auxiliary components in the 2D images 410. Labels may be in the form of bounding boxes, labeled segmentation information, and so on. Labels are sometimes referred to, herein, as aAttorney Docket No.: 28510.983 (L0820PCT)target output of an item of training data. In some embodiments, the 2D images 410 may include or may be similar to the 2D dentition image 200 of FIG.2A or the 2D dentition image 250 of FIG.2B. The 3D models 415 may include or may be similar to the 3D dentition model 300 of FIG.3.

[0156] In one embodiment, the training data labeler 434 uses the 3D models 415 to confirm, modify, add, etc. labels to the 2D images 410 with respect to auxiliary components. For example, the training data labeler 434 may obtain, as input, a 2D image 410 and a 3D model 415 that corresponds to the 2D image 410. As discussed above, each 2D image 410 and 3D model 415 may include visual features corresponding to auxiliary components disposed on a patients dentition. The training data labeler 434 may use the auxiliary component detector 112 to register the 3D model 415 to the 2D image 410 and correspond the one or more auxiliary components depicted in the 3D model 415 with the one or more auxiliary components depicted in the 2D image 410 based at least in parton information from the 3D model 415, as discussed herein. The training data labeler 434 may add a label to the 2D image 410 at locations in the 2D image 410 corresponding to the auxiliary components. The label may be a bounding box (e.g., a bounding box 252, as depicted in the 2D dentition image 250 of FIG.2B) or other data indicating the location of an auxiliary component. The label may include additional auxiliary component data, such as a tooth label, a component type of an auxiliary component, or other auxiliary component information. The training data labeler 434 may repeat this process to generate multiple items of training data. By registering the 3D model 415 to the corresponding 2D image 410, the training data labeler 434 can automatically generate training data for the auxiliary component detection model, and the training data may be more accurate than training data labeled by a human or by other systems.

[0157] In one embodiment, the training data labeler 434 may remove or modify incorrectly labeled training data items from a training dataset for the auxiliary component detection model. The training data labeler 434 may determine that an item of a training dataset has been incorrectly labeled by obtaining the label of an item of the training dataset. The auxiliary component detector 112 may obtain a 3D model 415 that corresponds to the 2D image 41 O of the training input of the item. The training data labeler 434 may use the auxiliary component detector 112 to register the 3D model 415 to the 2D image 410 included in the item’s training input. The auxiliary component detector 112 may then compare the locations of the auxiliary components indicated by the label with the locations of the expected auxiliary components of the corresponding registered 3D model 415. The comparison may indicate that the locations of the auxiliary components of the label do not match the locations of the expected auxiliary components. For example, a location of an auxiliary component of the label may not correspond to an expected auxiliary component of the 3D model 415, which may indicate a false positive. In another example, an expected auxiliary component of the 3D model 415 may notAttorney Docket No.: 28510.983 (L0820PCT)correspond to any of the locations indicated by the label, which may indicate a false negative. In response to the label not matching the expected auxiliary components, the auxiliary component detector 112 may determine that the item of the training dataset is incorrectly labeled. In response, the training data labeler 434 may remove the item from the training dataset 436, or the training data labeler 434 may modify the label so the locations of the auxiliary components of the label match the locations of the expected auxiliary components as indicated by the corresponding 3D model 415.

[0158] Processing logic may gather a training dataset 436 comprising 2D images of dental arches. One or more images and optionally associated probability maps in the training dataset 436 may be resized in embodiments. For example, a machine learning model may be usable for images having certain pixel size ranges, and one or more images may be resized if they fall outside of those pixel size ranges. The images may be resized, for example, using methods such as nearest-neighbor interpolation or box sampling. The training dataset may additionally or alternatively be augmented. Training of large-scale neural networks generally uses tens of thousands of images, which are not easy to acquire in many real-world applications. Data augmentation can be used to artificially increase the effective sample size. Common techniques include random rotation, shifts, shear, flips and so on to existing images to increase the sample size.

[0159] To effectuate training 438, processing logic inputs the training dataset(s) 436 into one or more untrained machine learning models. Prior to inputting a first input into a machine learning model, the machine learning model may be initialized. Processing logic trains the untrained machine learning model(s) based on the training dataset(s) to generate one or more trained machine learning models that perform various operations as set forth above.

[0160] Training may be performed by inputting one or more of the 2D images of the training dataset into the machine learning model one at a time. Each input may include data from an image in a training data item from the training dataset 436. The training data item may include, for example, a 2D image of a patients dentition.

[0161] The machine learning model processes the input to generate an output. An artificial neural network includes an input layer that consists of values in a data point (e.g., intensity values and / or height values of pixels in a height map). The next layer is called a hidden layer, and nodes at the hidden layer each receive one or more of the input values. Each node contains parameters (e.g., weights) to apply to the input values. Each node therefore essentially inputs the inputvalues into a multivariate function (e.g., a non-linear mathematical transformation) to produce an output value. A next layer may be another hidden layer or an output layer. In either case, the nodes at the next layer receive the output values from the nodes at the previous layer, and each node applies weights to those values and then generates its own output value. This may be performed at each layer. A final layer is theAttorney Docket No.: 28510.983 (L0820PCT)output layer, where there is one node for each class, prediction and / or output that the machine learning model can produce. For example, for an artificial neural network being trained to perform dental site classification, there may be a first class (dental auxiliary), a second class (teeth), a third class (gums), and / or one or more additional dental classes. Moreover, the class, prediction, etc. may be determined for each pixel in the image / scan / surface, may be determined for an entire image, or may be determined for each region or group of pixels of the image / scan / surface. For pixel level segmentation, for each pixel in the image, the final layer applies a probability that the pixel of the image belongs to the first class, a probability that the pixel belongs to the second class, a probability that the pixel belongs to the third class, and / or one or more additional probabilities that the pixel belongs to other classes. In some cases, the ML model may be trained to determine bounding boxes around dental auxiliary components.

[0162] Accordingly, the output may include one or more prediction, one or more a probability map and / or one or more bounding boxes. For example, an output probability map may comprise, for each pixel in an input image, a first probability that the pixel belongs to a first dental class, a second probability that the pixel belongs to a second dental class, and so on. For example, the probability map may include probabilities of pixels belonging to dental classes representing a tooth, gingiva, ora dental auxiliary. In further embodiments, different dental classes may represent different types of restorative objects.

[0163] Processing logic may then compare the generated probability map, bounding box and / or other output to the known probability map, bounding box and / or label that was included in the training data item. Processing logic determines an error (i.e., a classification error) based on the differences between the output probability map, bounding box and / or label (s) and the provided probability map, bounding box and / or label (s). Processing logic adjusts weights of one or more nodes in the machine learning model based on the error. An error term or delta may be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters for one or more of its nodes (the weights for one or more inputs of a node). Parameters may be updated in a back propagation manner, such that nodes at a highest layer are updated first, followed by nodes at a next layer, and so on. An artificial neural network contains multiple layers of “neurons”, where each layer receives as input values from neurons at a previous layer. The parameters for each neuron include weights associated with the values that are received from each of the neurons at a previous layer. Accordingly, adjusting the parameters may include adjusting the weights assigned to each of the inputs for one or more neurons atone or more layers in the artificial neural network.

[0164] Once the model parameters have been optimized, model validation may be performed to determine whether the model has improved and to determine a current accuracy of the deep learning model. After one or more rounds of training, processing logic may determine whether a stoppingAttorney Docket No.: 28510.983 (L0820PCT)criterion has been met. A stopping criterion may be a target level of accuracy, a target number of processed images from the training dataset, a target amount of change to parameters over one or more previous data points, a combination thereof and / or other criteria. In one embodiment, the stopping criteria is met when at least a minimum number of data points have been processed and at least a threshold accuracy is achieved. The threshold accuracy may be, for example, 70%, 80% or 90% accuracy. In one embodiment, the stopping criteria is met if accuracy of the machine learning model has stopped improving. If the stopping criterion has not been met, further training is performed. If the stopping criterion has been met, training may be complete, and the machine learning model may be stored in an Al model storage 445. Once the machine learning model is trained, a reserved portion of the training dataset may be used to test the model.

[0165] In one embodiment, model application workflow 417A includes one or more trained machine learning models (e.g., from the Al model storage 445) used by an auxiliary component detector 470. For model application workflow 417, according to one embodiment, processing logic receives one or more 2D images 448 and determines an appropriate 3D model 455 corresponding to a current stage of dental treatment for a patient. The 2D image(s) 448 and 3D model 455 may then be input into auxiliary component detector 470. In some embodiments, the 2D image(s) 448 may include or may be similar to the 2D dentition image 200 of FIG.2A or 2B. The 3D model 455 may include or may be similar to the 3D dentition model 300 of FIG.3.

[0166] Auxiliary component detector 470 may include one or more Al models (e.g., the auxiliary component detection model, discussed above, or one or more other ML models) and / or additional logic for identifying missing dental auxiliary components in 2D images 448 (e.g., one or more of the auxiliary components 204 of FIG.2A or FIG.2B).

[0167] As an example, in one embodiment, a machine learning model of auxiliary component detector 470 is trained to segment images by classifying regions of those intraoral images into one or more dental classes and / or determining bounding boxes around dental auxiliary components. In one embodiment, a first ML model is trained to perform segmentation of 2D images 448 of dentition into individual teeth and gingiva, and a second ML model is trained to receive segmentation information of a 2D image 448 output by the first ML model and to determine bounding boxes around detected dental auxiliar components.

[0168] As another example, in some embodiments, a machine learning model may determine one or more locations of the auxiliary components in a 2D image 448. The machine learning model may be an image recognition Al model trained and configured to detect auxiliary components in an input image. The machine learning model may use the 2D image 448 as input and may perform an inference calculation. The machine learning model can generate an output indicating one or more locations ofAttorney Docket No.: 28510.983 (L0820PCT)auxiliary components in the 2D image 448. The output may include data indicating a bounding box around each identified auxiliary component. In some embodiments, expected locations of dental auxiliary components are input into the machine learning model together with the 2D image 448. In some embodiments, the machine learning model may be trained to focus on locations at which auxiliary components are expected (e.g., from input expected location information), and to output, for each such location, whether an auxiliary component was detected.

[0169] The auxiliary component detector 470 may obtain one or more target locations of the expected auxiliary components from the 3D model 415, which the auxiliary component detector 470 registered to the 2D image 448. The registered 3D model 455 may include data indicating a location of each expected auxiliary component. The location data may include a coordinate in a 3D coordinate system used by the 3D model 455 projected onto the 2D image 448. The auxiliary component detector 470 may match the one or more expected auxiliary components to the one or more detected auxiliary components of the 2D image 448 by comparing the one or more target locations of the expected auxiliary components with the one or more locations of the auxiliary components in the 2D image 448. For example, the auxiliary component detector 470 may determine if each target location is contained within a bounding box of the 2D image 448 and that each bounding box contains only one target location. Based on such matching, the auxiliary component detector 470 may identify locations at which auxiliary components were expected but were not detected, which may be missing auxiliary components. In another example, the one or more locations of the auxiliary components in the 2D image 448 may be projected onto the 3D model 455. If no location of an auxiliary component is within a threshold distance from a target location, then the auxiliary component detector 470 may determine that the expected auxiliary component corresponding to the target location is not present in the 2D image 448. In cases where the auxiliary component is not detected or determined to be present, the system (e.g., the auxiliary component detector 470) may output missing auxiliary information 475 indicating that there are missing auxiliary components. The missing auxiliary information 475 may specify the expected location or tooth number of each missing auxiliary component.

[0170] Action determiner 476 may determine one or more actions to perform based on missing auxiliary information 475. Actions may include, for example, generating a notification 478 of the one or more missing dental auxiliary components, generating a modified 2D image highlighting the missing dental auxiliary components, outputting the notice / modified 2D image to a display of a patient device and / or dental professional device, recommending reattachment of a missing dental auxiliary component, recommending an in-person visit of the patient to the dental professional, scheduling such an in-person visit, ordering replacement dental auxiliary components, and so on. In some embodiments, action determiner 476 determines whether any missing auxiliary components have clinical significanceAttorney Docket No.: 28510.983 (L0820PCT)and then determines an action to perform based on whether or not any missing auxiliary component has clinical significance. For example, a patient may have 6 attachments bonded to their teeth at the beginning of aligner treatment, each for achieving certain movements that are scheduled to happen at different stages of the treatment. In this example, a first attachment may have been bonded to a tooth to assist with rotating the tooth by a certain amount during particular stages of the treatment (e.g., stages 5-9) and may not be required after that rotation is completed. If the attachment is determined to be missing after the rotation is completed (e.g., during stage 11), the attachment may no longer be necessary, and the action determiner 476 may determine that there is no clinical significance to the attachment being missing. As another example, a patient may have started a combination treatment that includes a palatal expansion treatment phase with a palatal expander to expand the patients palate followed by an aligner treatment phase to align the patients teeth. In this example, the patient may have had attachments placed on distal molars to retain the palatal expander, and these attachments may not be required for the subsequent aligner treatment. Thus, if the attachment is determined to be missing during the aligner treatment phase, the action determiner 476 may determine that there is no clinical significance to the attachment being missing.

[0171] If a missing dental auxiliary component does have clinical significance, then this fact may be indicated in a generated notification. If a missing dental auxiliary component does have clinical significance, then a recommendation for the missing dental auxiliary component to be replaced may be generated. If no missing dental auxiliary component has clinical significance, then such fact may be indicated in an output notification. If no missing dental auxiliary component has clinical significance, then there may be no need to replace the missing dental auxiliary component, and treatment may proceed as planned without reattaching the missing dental auxiliary component. The notifications and / or recommendations may be sent to a device of the patient and / or an associated dental professional.

[0172] FIG.4B illustrates a model training workflow 405B and a model application workflow 417B for a dental treatment computing system 480 capable of detecting tooth attachments and other auxiliaries using per-tooth processing, in accordance with an embodiment of the present disclosure. The model training workflow 405B and the model application workflow 417B may be performed by processing logic executed by a processor of a computing device. One or more of these workflows 405B, 417B may be implemented, for example, by one or more machine learning modules implemented in dental treatment computing system 510 of FIG.5.

[0173] The model training workflow 405B trains one or more machine learning models to perform classification tasks for cropped 2D images of individual teeth and / or auxiliary components in embodiments. The model application workflow 417B is to apply the one or more trained machineAttorney Docket No.: 28510.983 (L0820PCT)learning models to perform classification tasks on cropped per-tooth images and / or per-auxiliary component images to identify auxiliary components on individual teeth, and to perform one or more actions in response to detecting missing auxiliary components.

[0174] For the model training workflow 405B, processing logic may gather 2D images 410 and optionally 3D models 415 as inputs. The 2D images 410 may include images of patients' dentitions captured by patient devices and / or dental professional devices (e.g., such as intraoral scanning systems). The 3D models 415 may include 3D dentition models associated with treatment plans for the patients. In one embodiment, a training data labeler 434 receives the 2D images 410 and 3D models 415 and processes them to generate labeled training data. The training data labeler 434 may use the 3D models 415 to confirm, modify, or add labels to the 2D images 410 with respect to auxiliary components, as discussed above with reference to FIG.4A.

[0175] The 2D images 410 are provided to a tooth seg menter / cl assifier 435, which performs segmentation and / or classification operations on the input images to identify individual teeth. The tooth seg menter / cl assifier 435 may include one or more Al models trained to perform object detection to detect teeth, wherein the Al models output bounding boxes around each of the teeth in the 2D images 410. Alternatively or additionally, the tooth segmenter / classifier 435 may include one or more Al models trained to perform segmentation of teeth in images, wherein the Al models output segmentation information for the teeth in the 2D images 410. The segmentation information may include pixel-level classifications indicating which pixels belong to which teeth.

[0176] The output from the tooth segmenter / classifier 435 is provided to an image cropper 437. The image cropper 437 generates cropped images based on the segmentation information and / or bounding box information from the tooth segmenter / classifier 435. Each cropped image is associated with a tooth and includes pixels of the 2D image for the tooth while excluding some or all pixels for a remainder of the teeth. In some embodiments, the image cropper 437 may identify, for a cropped image and based on the segmentation information, one or more pixels of the cropped image that do not belong to the tooth associated with the cropped image, and may modify the cropped image by setting values for those pixels to a predetermined value (e.g., black), wherein the pixels having the predetermined value may be ignored by classification models used fortraining and inference.

[0177] In some embodiments, the model training workflow 405B may integrate the 3D model registration techniques described with reference to FIG. 4A. For example, the training data labeler 434 may register a 3D model 415 to a corresponding 2D image 410 to determine locations of auxiliary components. The image cropper 437 may then generate cropped images that are focused on specific locations where auxiliary components are located rather than on entire teeth. In such embodiments, a cropped image may be associated with a tooth and an auxiliary component on the tooth, and theAttorney Docket No.: 28510.983 (L0820PCT)cropped image may include pixels of the 2D image for the associated auxiliary component and exclude at least some pixels of the 2D image for the tooth. This per-auxiliary-component cropping approach may provide higher resolution images focused specifically on the regions where auxiliary components are expected.

[0178] The image cropper 437 outputs cropped images, which may be used to assemble a training dataset 436. The labeled cropped images of the training dataset 326 may be stored in a data store in embodiments. Each item in the training dataset 436 may include a cropped image as a training input and a label specifying whether the cropped image includes an auxiliary component and / or a location of the auxiliary component in the image. If an Al model is to be trained to perform image classification, then location information of the auxiliary component may not be used. If an Al model is to be trained to perform object detection or segmentation associated with auxiliary components, then the location information (e.g., a bounding box and / or pixel map identifying pixels for the auxiliary component) may be included. The labels may be determined through registration of 3D models 415 to the 2D images 410, through manual labeling, or through a combination of both approaches. In some embodiments, the labels may further specify the type of auxiliary component (e.g., attachment, button, bracket, power ridge) present in the cropped image.

[0179] The training dataset 436 is used to train one or more ML model(s) by Al model trainer 438. The one or more machine learning models may include one or more classification models trained to perform image classification. In some embodiments, the one or more classification models may include a single model that classifies multiple types of auxiliary components. In other embodiments, the one or more classification models may include a plurality of classification models, each trained to perform classification for a different type of auxiliary component, such as a first model that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and a fourth model that identifies brackets. The trained models are stored in an Al model storage 445. The various classification models may be or include one or more neural networks in embodiments, such as convolutional neural networks, deep neural networks, and so on.

[0180] Once the Al model(s) are trained, they may be used in model application workflow 417B, such as by auxiliary component detector 470. For the model application workflow 417B, processing logic receives one or more 2D dentition images 448 . The 2D dentition images 448 may include images of a patient's dentition captured by a patient device (e.g., a camera of a mobile phone of a patient) or dental professional device. The 2D images 448 are processed by the tooth segmenter / classifier 435, which identifies and segments individual teeth in the images. The tooth segmenter / classifier 435 may output bounding boxes around each of the teeth in the 2D images 448 and / or segmentation information for the teeth.Attorney Docket No.: 28510.983 (L0820PCT)

[0181] The outputfrom the tooth segmenter / classifier 435 (e.g., bounding boxes around teeth and / or segmentation information indicating the pixels associated with each of the teeth, optionally provided as one or more segmentation masks) is provided to the image cropper 437, which generates cropped images for each identified tooth. Each cropped image is associated with a tooth and includes pixels of the 2D image 448 for the tooth while excluding some or all pixels for a remainder of the teeth. If the output of the segmenter / classifier 435 includes object detection information (e.g., such as bounding boxes for each of the teeth in the 2D dentition image 448), then such object detection information may be used to set the boundaries to use in cropping the 2D dentition image(s) 448 to generate cropped images. If the output of the segmenter / classifier 435 includes segmentation information, then bounding boxes may be determined for each of the teeth based on the segmentation information. Each of the determined bounding boxes may encapsulate one of the segmented teeth. In some embodiments, the image cropper 437 may modify the cropped images by setting values for pixels that do not belong to the associated tooth to a predetermined value, such as black. Output cropped images may be provided to auxiliary component detector 470 in embodiments.

[0182] In some embodiments, the model application workflow 417B may receive additional treatment plan data 439 as an input to the image cropper 437 along with the output of the tooth segmenter / classifier 435. The treatment plan data 439 may include information identifying which teeth should have auxiliary components attached thereto and / or the types of auxiliary components that respective teeth should have. The treatment plan data 439 may also include a 3D model of the patient's dentition, such as the 3D model 455, which indicates expected locations and configurations of auxiliary components. The image cropper 437 may use the treatment plan data 439 to generate cropped images of individual auxiliary components rather than entire teeth, enabling more focused analysis of specific locations where auxiliary components are expected. The cropped images of individual auxiliary components generated by the image cropper 437 based on the treatment plan data 439 may then be provided to the auxiliary component detector 470 for classification and detection operations.

[0183] In an example, the model application workflow 417B may integrate the 3D model registration techniques described with reference to FIG. 4A. A 3D model 455 corresponding to a current stage of dental treatment for the patient may be obtained and registered to the 2D image 448. Based on the registration, expected locations of auxiliary components may be determined. The image cropper 437 may then generate cropped images that are focused on specific locations where auxiliary components are expected. In such embodiments, rather than generating one cropped image per tooth, the image cropper 437 may generate one cropped image per expected auxiliary component. If a tooth has multiple expected auxiliary components (e.g., both an attachment and a button), multiple croppedAttorney Docket No.: 28510.983 (L0820PCT)images may be generated for that tooth, each focused on a different expected auxiliary component location.

[0184] The cropped images from the image cropper 437 are provided to an auxiliary component detector 470. The auxiliary component detector 470 utilizes trained models from the Al model storage 445 to detect auxiliary components in the cropped images. For each processed cropped image, the trained models output an indication of whether the tooth or expected auxiliary component location associated with the cropped image has one or more attached auxiliary components.

[0185] In some embodiments, the auxiliary component detector 470 may determine, based on a treatment plan or the 3D model 455, one or more types of expected auxiliary components for each tooth. The auxiliary component detector 470 may then determine which classification models ofa plurality of classification models to apply to each cropped image based on the types of expected auxiliary components. For example, if the treatment plan indicates that a particular tooth should have an attachment but not a button, the auxiliary component detector 470 may apply an attachment classification model to the cropped image for that tooth but may not apply a button classification model. This selective application of models may reduce computational overhead and may reduce false positives.

[0186] In embodiments where multiple classification models are applied to a cropped image (or one or more classification models that can classify multiple types of auxiliary components are applied to the cropped image), the auxiliary component detector 470 may receive a plurality of confidence values, each associated with a different type of auxiliary component. The auxiliary component detector 470 may determine which types of auxiliary components are present based on whether the confidence values satisfy one or more criteria, such as exceeding a confidence threshold or being greater than confidence values associated with other types of auxiliary components. In some cases, disambiguation logic may be applied to resolve situations where multiple auxiliary component types have confidence values that satisfy the criteria. The disambiguation logic may be rule-based (e.g., determining that certain types of auxiliary components cannot coexist on the same tooth) or probabilistic (e.g., applying different confidence thresholds based on the rarity of certain auxiliary component types), or combinations thereof.

[0187] The auxiliary component detector 470 may also receive the 3D model 455 and compare the detected auxiliary components against expected auxiliary components indicated by the 3D model 455. Based on this comparison, the auxiliary component detector 470 may identify auxiliary components that are expected but not detected, which may be missing auxiliary components. The auxiliary component detector 470 may output missing auxiliary information 475, which identifies any auxiliary components that are expected but not detected in the 2D image(s) 448. The missing auxiliaryAttorney Docket No.: 28510.983 (L0820PCT)information 475 may specify the expected location or tooth number of each missing auxiliary component. The auxiliary component detector 470 may also output information identifying auxiliary components that were detected at expected locations, that were detected as unexpected locations, and / or that were not detected at locations where no auxiliary components were expected.

[0188] The missing auxiliary information 475 and / or other output information is provided to an action determiner 476, which determines appropriate actions based on the detected missing components and / or other output information. The action determiner 476 may determine whether any missing auxiliary components have clinical significance and may determine actions to perform based on whether or not any missing auxiliary component has clinical significance, as discussed above with reference to FIG.4A.

[0189] In embodiments, the action determiner 476 generates a notification 478 to alert users of any missing auxiliary components. The notification 478 may include various types of outputs depending on the nature and severity of the detected missing auxiliary components. For example, the notification 478 may include a recommendation for the patient to schedule an appointment with a dental professional to have the missing auxiliary component reattached or replaced. In some embodiments, the notification 478 may include instructions for the patient to discontinue use of a current dental appliance until the missing auxiliary component is addressed, particularly when the auxiliary component is critical for proper appliance function or treatment efficacy. The notification 478 may include an urgency indicator specifying whether immediate attention is required or whether the patient can continue treatment until a regularly scheduled appointment. In some embodiments, the notification 478 may include information identifying the specific tooth from which the auxiliary component is missing, the type ofauxiliarycomponentthat is missing, and / or an estimated impact on treatment progress if the missing component is not replaced. The notification 478 may also include alternative treatment recommendations, such as suggestions to skip certain aligners in a treatment sequence or to extend wear time of a current aligner until the auxiliary component can be reattached. In some embodiments, the action determiner 476 may generate different notifications for different recipients, such as a first notification sent to the patient device with patient-friendly instructions and a second notification sent to the dental professional device with detailed clinical information to facilitate efficient reattachment during a subsequent appointment.

[0190] In some embodiments, the model application workflow 417B may be used for longitudinal monitoring of auxiliary components. Processing logic may obtain a new 2D dentition image at a second time that is later than a first time at which an initial 2D dentition image was obtained. The new 2D dentition image may be processed through the tooth segmenter / classifier 435, image cropper 437, and auxiliary component detector 470 to generate new indications of which teeth have attached auxiliaryAttorney Docket No.: 28510.983 (L0820PCT)components. A comparison may be made between the new indications and indications generated from processing of the initial 2D dentition image to identify any differences, such as auxiliary components that were present in the initial image but are missing from the new image. A notice of the differences may be output to alert a dental professional orthe patient to potential issues.

[0191] In some embodiments, the model application workflow 417B may operate without treatment plan data 439 or a 3D model 455. This configuration may be applicable in scenarios where a treatment plan is not available, such as for patients undergoing wire and bracket orthodontic treatment rather than aligner-based treatment. In such embodiments, prior auxiliary component detection results may be used as a baseline for comparison with current detection results. For example, at a start of treatment or at an initial monitoring session, a first 2D dentition image may be captured and processed through the tooth segmenter / classifier 435, image cropper 437, and auxiliary component detector 470 to generate initial indications of which teeth have attached auxiliary components and the types of auxiliary components presenton each tooth. These initial detection results may be stored and associated with the patient record. The stored initial detection results may serve as a ground truth representing the expected state of the patient's dentition for subsequent monitoring sessions. When a new 2D dentition image is obtained at a later time, the new image may be processed through the model application workflow 417B to generate new indications of which teeth have attached auxiliary components. The auxiliary component detector 470 may compare the new indications against the stored initial detection results rather than against treatment plan data or a 3D model. Based on this comparison, the auxiliary component detector 470 may identify any differences between the initial detection results and the new detection results, such as auxiliary components that were detected in the initial image but are not detected in the new image. Such differences may indicate that an auxiliary component has become debonded or damaged since the initial monitoring session. The missing auxiliary information 475 may then identify the auxiliary components that were previously detected but are now missing, enabling the action determiner 476 to generate appropriate notifications 478 alerting the dental professional or patient to the potential issue. This approach may enable monitoring of patients for missing auxiliary components even when detailed treatment plan information is not available, by treating the initial detected state as the baseline against which subsequent images are compared.

[0192] FIG. 5 illustrates an example system architecture 500 for virtual dental care which is capable of auxiliary component detection, in accordance with one embodiment of the present disclosure. The system 500 can include a dental treatment computing system 510, a patient device 520, a dental professional device 530, and / or a computer network 540. In some embodiments, the dental treatment computing system 510 generally represents any type or form of computing device thatAttorney Docket No.: 28510.983 (L0820PCT)is capable of storing and analyzing data. The dental treatment computing system 510 may include a backend database server for storing patient data and treatment data (e.g., the dental treatment datastore 514). Additional examples of the dental treatment computing system 510 include, without limitation, security servers, application servers, web servers, storage servers, and / or database servers configured to run certain software applications and / or provide various security, web, storage, and / or database services. Although illustrated as a single entity in FIG. 5, the dental treatment computing system 510 may include and / or represent a plurality of servers that work and / or operate in conjunction with one another. In embodiments, a dental treatment computing system 510 can support any number of discrete software or hardware, or combination of such, portions, which can be referred to as subsystems or modules.

[0193] In one embodiment, the dental treatment computing system 510 can include an auxiliary component detector 470. The auxiliary component detector 470 may include the auxiliary component detector 470 of FIG. 4A and / or FIG. 4B, discussed above. In some embodiments, the auxiliary component detector 470 may detect whether any auxiliary components of a patient are missing in a 2D dentition image 448 based at least in parton information from a 3D dentition model 455 associated with a patient. In some embodiments, the auxiliary component detector 470 may detect auxiliary components using per-tooth processing, wherein the 2D dentition image 448 is segmented to identify individual teeth, cropped 2D images are generated for each tooth, and the cropped 2D images are processed using one or more classification models to determine whether auxiliary components are presenton each tooth. In some embodiments, the auxiliary component detector 470 may combine 3D dentition model registration with per-tooth or per-auxiliary-component processing to enable focused analysis on specific locations where auxiliary components are expected. The dental treatment computing system 510 can also include a dental treatment datastore 514. The dental treatment datastore 514 can store data used by the dental treatment computing system 510 and, specifically, by the auxiliary component detector 470. The datastore 514 can store 2D dentition images 515 and 3D dentition models 516.

[0194] In some embodiments, the patient device 520 can include a computing device (e.g., a mobile computing device, a personal computer, etc.) with a camera 522. As an example of various implementations, a camera 522 of the patient device 520 may be an ordinary camera that captures 2D images. In some examples, the patient device 520 may include an at-home intraoral scanner.

[0195] In some implementations, the patient device is configured to interface with a dental consumer and / or dental patient. A "dental consumer," as used herein, may include a person seeking assessment, diagnosis, and / or treatment for a dental condition (general dental condition, orthodontic condition, endodontic condition, condition requiring restorative dentistry, etc.). A dental consumer may,Attorney Docket No.: 28510.983 (L0820PCT)but need not, have agreed to and / or started treatment for a dental condition. A "dental patient," as used herein, may include a person who has agreed to diagnosis and / or treatment for a dental condition. A dental consumer and / or a dental patient, may, for instance, be interested in and / or have started orthodontic treatment, such as treatment using one or more (e.g., a sequence of) aligners (e.g., polymeric appliances having a plurality of tooth-receiving cavities shaped to successively reposition a person's teeth from an initial arrangement toward a target arrangement). In various implementations, the patient device 520 provides a dental consumer / dental patient with software (e.g., one or more webpages, standalone applications, mobile applications, etc.) that allows the dental consumer / patient to capture images of their dentition, interact with dental professionals (e.g., users of the dental professional device 530) and / or manage treatment plans (e.g., those from the dental treatment computing system 510 and / or the dental professional device 530).

[0196] The patient device 520 may include software (e.g., a mobile application, web browser executing script of a website, etc.) that obtains a 2D dentition image 448 captured by the camera 522 and provides the 2D dentition image 448 to the dental treatment computing system 510. A patient associated with the patient device 520 may use the camera to capture a 2D dentition image 448 of the patient's mouth (e.g., dentition, oral cavity, smile, etc.) so the dental treatment computing system 510 can process the image to make one or more assessments about treatment progress. In embodiments, the dental treatment computing system 510 is configured to determine whether any of the auxiliary components that should be attached to the patient's dentition are missing.

[0197] The dental professional device 530 generally represents any type or form of computing device capable of reading computer-executable instructions. The dental professional device 530 may be, for example, a desktop computer, a tablet computing device, a laptop, a smartphone, an augmented reality device, or other consumer device.

[0198] In various implementations, the dental professional device 530 is configured to interface with a dental professional. A "dental professional" (used interchangeably with dentist, orthodontist, and doctor) as used herein, may include any person with specialized training in the field of dentistry, and may include, without limitation, general practice dentists, orthodontists, dental technicians, dental hygienists, etc. A dental professional may include a person who can assess, diagnose, and / or treat a dental condition. "Assessment" of a dental condition, as used herein, may include an estimation of the existence of a dental condition. An assessment of a dental condition need not be a clinical diagnosis of the dental condition. In some embodiments, an "assessment1of a dental condition may include an "image based assessment," that is an assessment of a dental condition based in part or on whole on photos and / or images (e.g., images that are not used to stitch a mesh or form the basis of a clinical scan) taken of the dental condition. A "diagnosis" of a dental condition, as used herein, may include aAttorney Docket No.: 28510.983 (L0820PCT)clinical identification of the nature of an illness or other problem by examination of the symptoms. "Treatment" of a dental condition, as used herein, may include prescription and / or administration of care to address the dental conditions. Examples of treatments to dental conditions include prescription and / or administration of brackets / wires, clear aligners, palatal expanders, and / or other appliances to address orthodontic conditions, prescription and / or administration of restorative elements to address bring dentition to functional and / or aesthetic requirements, etc. The dental professional device 530 may provide to a dental practitioner software (e.g., one or more webpages, standalone applications (e.g., dedicated treatment planning and / or treatment visualization applications), mobile applications, etc.) that allows the dental practitioner to interact with patients, other dental professionals, etc., to create / modify / manage treatment plans (e.g., those from the dental treatment computing system 510 and / or those generated at the dental professional device 530), etc.

[0199] In one embodiment, the dental professional device 530 may include a computing device that includes software that obtains data from the dental treatment computing system 510. The obtained data may include data indicating whether one or more expected auxiliary components of the patient associated with the patient device 520 is missing. The user interface (III) 532 of the dental professional device 530 may present information to the device's 530 user regarding the missing auxiliary component. A dental professional associated with the dental professional device 530 can then assist the patient in replacing or otherwise remedying the missing auxiliary component.

[0200] In some embodiments, network 540 can connectthe various platforms and / or devices, which can include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network ora Wi-Fi network), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or a combination thereof.

[0201] Some embodiments provide patients with "Virtual dental care." "Virtual dental care," as used herein, may include computer-program instructions and / or software operative to provide remote dental services by a health professional (dentist, orthodontist, dental technician, etc.) to a patient, a potential consumer of dental services, and / or other individual. Virtual dental care may comprise computer-program instructions and / or software operative to provide dental services without a physical meeting and / or with only a limited physical meeting. As an example, virtual dental care may include software operative to providing dental care from the dental professional device 530 and / or the dental treatment computing system 510 to the patient device 520 over the network 540 through e.g., written instructions, interactive applications that allow the health professional and patient / consumerto interact with one another, telephone, chat etc. Some embodiments provide patients with "Remote dental care." "Remote dental care," as used herein, may comprise computer-program instructions and / or softwareAttorney Docket No.: 28510.983 (L0820PCT)operative to provide a remote service in which a health professional provides a patient with dental health care solutions and / or services. In some embodiments, the virtual dental care facilitated by the elements of the system 500 may include non-clinical dental services, such as dental administration services, dental training services, dental education services, etc. In some embodiments, dental treatment computing system 510 provides virtual care in accordance with the disclosure of U.S. Patent Application No. 18 / 976,167, filed December 10, 2024, which is incorporated by reference herein in its entirety.

[0202] In some embodiments, the system architecture 500 may be used to facilitate virtual dental care during orthodontic treatment. Virtual dental care may involve providing dental services remotely through digital communication channels, enabling dental professionals to monitor patient progress, assess treatment efficacy, and provide guidance without requiring the patient to physically visit a dental office. The patient device 520 may capture 2D dentition images using a camera 522 of patient device 520, and these images may be transmitted to the dental treatment computing system 510 via the network 540 for analysis. The dental treatment computing system 510 may process the received images using an auxiliary component detector 470 to determine whether expected auxiliary components are presenton the patient's teeth, whether any auxiliary components are missing or damaged, and whether the patient's treatment is progressing as planned. Results of this analysis may be provided to the dental professional device 530, where a dental professional may review the findings and determine whether any intervention is needed. This remote monitoring capability may enable dental professionals to track multiple patients' treatment progress without requiring each patient to visit the office for routine check-ups.

[0203] The auxiliary component detection capabilities described herein may reduce the number of in-person visits required during orthodontic treatment. In traditional orthodontic care, patients may be required to visit a dental office at regular intervals (e.g., every few weeks) so that a dental professional can visually inspect the patient's dentition and verify that auxiliary components such as attachments and buttons remain properly bonded to the teeth. By enabling automated detection of missing or damaged auxiliary components from 2D images captured by the patient at home, the system architecture 500 may allow dental professionals to remotely verify the status of auxiliary components between scheduled appointments. If the auxiliary component detector 470 determines that all expected auxiliary components are present and properly positioned, the dental professional may determine that an in-person visit is not immediately necessary. Conversely, if the auxiliary component detector 470 identifies a missing auxiliary component that has clinical significance for the patient's ongoing treatment, the dental professional may be notified promptly and may schedule an in-person visit specifically to address the issue. This targeted approach to scheduling in-person visits may reduceAttorney Docket No.: 28510.983 (L0820PCT)unnecessary office visits while ensuring that clinically significant issues are addressed in a timely manner.

[0204] In some embodiments, the elements of the system 500 may be operative to provide intelligent photo guidance to a patient to take images relevant to virtual dental care using the camera 522 on the patient device 520. This may include, for example, providing one or more photo parameters to capture clinically relevant photos of a user. "Clinically relevant" photos, as used herein, may include images that represent the state of dental conditions in a consumer / patient's dentition. Clinically relevant photos may include photos that are sufficient to provide current position(s) and / or orientation(s) of the teeth in a consumer / patient's mouth. Examples of clinically relevant photos include photos that show all the teeth in a consumer / patient's arch; photos that show the shape of a consumer / patient's arch; photos that show locations of teeth that are missing, supernumerary, ectopic, etc. photos; photos that show malocclusions in a consumer / patient's arch (e.g., from front, left buccal, right buccal, and / or other various perspectives); photos that show locations of dental auxiliaries, etc. "Photo parameters," as used this context, may include parameters to define clinically acceptable criteria (e.g., clinically acceptable position(s) and / or clinically acceptable orientation(s) of teeth) in one or more photos. In embodiments, some photos may show a patient's dentition and a dental appliance worn over the patient's dentition, while some photos may show the patient's dentition without a dental appliance. Photo parameters can include a distance parameters, e.g., one that parametrizes a distance that a camera is relative to a consumer / patient's dentition; orientation parameters (e.g., those that parametrize orientations of photos taken of teeth); openness parameters of a photo of a consumer / patient's bite (e.g., whether a bite is open, closed, and / or a degree of openness of a bite); a dental appliance wear parameter of a photo of a consumer / patient's bite (e.g., whether a photo shows dental appliances, such as cheek retractors, aligners, etc. in a consumer / patient's mouth); camera parameters (brightness parameters of photos; contrast parameters of photos; exposure parameters of photos; etc.); tooth identifier parameters, e.g., those that parametrize the specific teeth in a photo, those taken from a treatment plan; etc.

[0205] The patient device 520 may include a user interface (Ul) of a patient-focused virtual dental care software or service that may use the one or more photo parameters to intelligently guide the consumer / patient to capture clinically relevant photos of their dentition. The patient device 520 may gather image-capture rules that guide capturing the clinically relevant photos based on the photo parameters. The patient device 520 may provide a consumer / patient with software (e.g., one or more webpages, standalone applications, mobile applications, etc.) that uses the one or more photo parameters to help the consumer / patient capture clinically relevant photos of their teeth. As an example, distance parameters may be used to guide a consumer / patient to position and / or orient the patient device 520 specific distance away from their teeth to capture a photo with appropriate details ofAttorney Docket No.: 28510.983 (L0820PCT)their teeth. The distance parameters may guide whether the position of a camera is too close or too far or just right. Orientation parameters may be used to guide a photo to clinically relevant orientations. As an example, orientation parameters may be used to guide a consumer / patient to take photos of anterior views, left buccal views, right buccal views, etc. As additional examples, openness parameters may be used to guide a consumer / patient to take photos of various bite states, e.g., an open bite, closed bite, and / or a bite that is partially open in order to be clinically relevant; dental appliance wear parameters may be used to detect cheek retractors and / or guide a consumer / patient to position cheek retractors appropriately and / or locate / orient photos to be clinically relevant; dental appliance wear parameters may be used to detect various dental appliances (aligners, retainers, etc.) and guide a consumer to remove, move, etc. the dental appliances for photos that are clinically relevant; etc. Additionally, tooth identifier parameters (e.g., those gathered from a treatment plan) can be used to guide a consumer / patient to take photos of a sufficient number of teeth so that the photos are clinically relevant. Camera parameters, e.g., contrast, brightness, exposure, etc. parameters may be used to guide consumers / patients to take photos that have properties such that the photos are clinically relevant. In some implementations, the patient device 520 uses camera parameters to modify one or more photo settings (add / disable flash, adjust zoom, adjust brightness, adjust contrast, adjust shadows, adjust silhouettes, etc. so that clinically relevant photos are captured under various conditions. In some embodiments, these operations may be performed by automated agents and without human intervention.

[0206] Patient device 520 may operate to capture clinically relevant photos using the intelligent guidance. In some implementations, a consumer / patient may follow instructions to capture photos of their dentition using the intelligent guidance provided on the patient device 520. In some embodiments, one or more 2D dentition images are captured while a patient wears a retraction device to retract the lips and expose the upper and lower teeth. In various implementations, one or more of the above operations is performed by automated agents that configure cameras 522 to take photos without human intervention.

[0207] Once patient device 520 captures the photos (e.g., clinically relevant images), it may send these photos to dental treatment computing system 510 and / or dental professional device 530 for analysis. In embodiments, dental treatment computing system 510 stores the photos in dental treatment datastore 514 (e.g., as dentition images 515). In some instances, dental treatment computing system 510 may send the photos to dental professional device 530 for assessment by a dental professional.

[0208] The dental treatment computing system 510 and / or dental professional device 530 may use the clinically relevant photos (e.g., dentition images 515) to assess patient progress with regards to a dental treatment plan. As an example, the dental treatment computing system 510 may process theAttorney Docket No.: 28510.983 (L0820PCT)clinically relevant photos to assess whether to advance to a subsequent stage of a dental treatment earlier than indicated by a treatment plan or to remain in a current stage of dental treatment longer than indicated by the treatment plan. In some implementations, the dental treatment computing system 510 may use clinically relevant photos for image-based assessments, intelligent patient guidance, and / or photo-based refinements of a dental treatment plan. The dental treatment computing system 510 may additionally or alternatively make other determinations with respect to a dental treatment plan and / or with respect to a patient's dentition based on analysis of the clinically relevant photos in embodiments. For example, dental treatment computing system 510 may identify one or more oral health conditions and / or a severity of such identified oral health conditions.

[0209] In embodiments, dental treatment computing system 510 may process dentition images 515 to identify missing dental auxiliary components (also referred to herein as dental auxiliaries) such as missing attachments and / or buttons.

[0210] In some embodiments, the elements of the system 500 (e.g., the patient device 520, dental professional device 530 and / or dental treatment computing system 510) may be operative to provide one or more image-based assessment tools to the users of the dental treatment computing system 510. "Image based assessment tools," as used herein, may include digital tools that operate to provide image-based assessments of a dental condition (e.g., such as missing dental auxiliaries). Such image-based assessments may be assessments of a dentition currently undergoing treatment and / or of a dentition for which treatment has not yet begun and / or for which treatment has been completed in embodiments. In some embodiments, image-based assessments may comprise visualizations that allow a user of the dental treatment computing system, dental professional device 530 and / or patient device 520 to make a decision about a clinical condition and / or progress of a treatment plan.

[0211] In some embodiments, the dental treatment computing system 510 can facilitate or host services for detecting auxiliary components in images ofa patient's dentition. In embodiments, the dental treatment computing system 510 can host, leverage, and / or include several subsystems or modules for supporting such system functionalities. For instance, in embodiments, the dental treatment computing system 510 can support and / or integrate a control module (not shown in FIG.5), for performing overall control of the subsystems, modules, and devices associated with the dental treatment computing system 510, and a III control module (not shown in FIG.5), for performing generation, and other processes associated with a III that will be presented through associated client devices 520, 530. The dental treatment computing system 510 can supporta data processing module (not shown in FIG.5), that can gather and manage data from data storage and other subsystems (such as 2D dentition images 448 or 3D dentition models 455 gathered from a datastore 514). The dental treatment computing system 510 can also process, transmit, and / or receive incoming and outgoing dataAttorney Docket No.: 28510.983 (L0820PCT)from client devices 520, 530. Such subsystems and modules can work collaboratively, and communicate internally or externally (e.g., to external systems and / or through APIs), to facilitate auxiliary component detection. Each subsystem and module can include hardware, firmware, and / or software configured to provide a described functionality.

[0212] In some embodiments, dental treatment computing system 510 (or an integrated control subsystem) can orchestrate the overall functioning of the dental treatment computing system 510. In some cases, the dental treatment computing system 510 can include algorithms and processes to direct the setup, data transfer, and processing for providing and receiving data associated with auxiliary component detection. For example, when a user initiates engagement with the dental treatment computing system 510, the system 510 can initiate and manage the associated processes, including allocating resources, determining routing pathways for data and data streams, managing permissions, and so forth to interact with client devices 520, 530 to establish and maintain reliable connections and data transfer.

[0213] The dental treatment computing system 510 can include a III controller and can perform user-display functionalities of the system such as generating, modifying, and monitoring the individual Ul(s) and associated components that are presented to users of the system 500 through a client device 520, 530. For example, a III control subsystem can generate the Ul(s) (e.g., the III 532 of the dental professional device 530) that users interact with while engaging with the dental treatment computing system 510.

[0214] A III can include many interactive (and / or non-interactive) visual elements for display to a user. Such visual elements can occupy space within a III and can be visual elements such as windows displaying video streams, windows displaying images, chat panels, file sharing options, participant lists, and / or control buttons for controlling functions such as client application navigation, file upload and transfer, controlling communications functions such as muting audio, disabling video, screen sharing, etc. The III control module can work to generate such a III, including generating, monitoring, and updating the spatial arrangement and presentation of such visual elements, as well as working to maintain functions and manage user interactions, together with the dental treatment computing system 510. Additionally, the III control module can adapt a user-interface based on the capabilities of client devices. In such a way the III control module can provide a fluid and responsive interactive experience for users of the treatment coordination platform.

[0215] In some embodiments, a data processing subsystem can be responsible for storage and management of data. This can include gathering and directing data from client devices 520, 530. In embodiments, the data processing subsystem can communicate and store data, including to and / or from storage platforms and storage devices (e.g., such as the dental treatment datastore 514), etc. ForAttorney Docket No.: 28510.983 (L0820PCT)instance, the auxiliary component detector 470 can receive 2D dentition images 448 and 3D dentition models 455 and use them to detect auxiliary components in a 2D dentition image 448.

[0216] In embodiments, the dental treatment computing system 510 can leverage the auxiliary component detector 470 for performing processes associated with detecting auxiliary components in 2D dentition images 448 provided by patients. The auxiliary component detector 470 may implement one or more approaches for detecting auxiliary components, including a first approach using 3D dentition model registration, a second approach using per-tooth processing, and a third approach combining 3D dentition model registration with per-tooth or per-auxiliary-component processing in embodiments.

[0217] In the first approach, the auxiliary component detector 470 may obtain a 2D dentition image 448 (e.g., from the dentition images 515) that depicts a patient's dentition (e.g., the dentition of the patient associated with the patient device 520). The auxiliary component detector 470 can obtain a 3D dentition model 455 (e.g., from the dentition models 516) that depicts the same patient's dentition. The 3D dentition model 455 may have been provided from the dental professional device 530 or another computing device associated with the orthodontic professional, and may be associated with or part of a treatment plan. For example, the 3D dentition model 455 may be a model (e.g., a 3D mesh) of the patient's dentition at a particular stage of dental treatment. The 3D dentition model 455 may indicate one or more expected auxiliary components, which may include auxiliary components attached to the patient's dentition by an orthodontic professional. The auxiliary component detector 470 may register the 3D dentition model 455 to the 2D dentition image 448 and determine, using the registered 3D dentition model 455 and the 2D dentition image 448, whether one or more expected auxiliary components are present in the 2D dentition image 448.

[0218] In the second approach, the auxiliary component detector 470 may perform per-tooth auxiliary component detection. The auxiliary component detector 470 may process the 2D dentition image 448 using one or more first models that output information identifying locations of a plurality of teeth in the 2D dentition image 448. The one or more first models may include an Al model trained to perform object detection to detect teeth, wherein the Al model outputs bounding boxes around each of the plurality of teeth in the 2D dentition image 448. Alternatively or additionally, the one or more first models may include an Al model trained to perform segmentation of teeth in images, wherein the Al model outputs segmentation information for the plurality of teeth in the 2D dentition image 448. The auxiliary component detector 470 may generate one or more cropped 2D images from the 2D dentition image 448 based on the information identifying the locations of the plurality of teeth, where each cropped 2D image is associated with a tooth and includes pixels of the 2D dentition image 448 for the tooth while excluding some or all pixels for a remainder of the plurality of teeth. The auxiliary component detector 470 may process the one or more cropped 2D images using one or more secondAttorney Docket No.: 28510.983 (L0820PCT)models, wherein for each processed cropped 2D image the one or more second models output an indication of whether the tooth associated with the cropped 2D image has one or more attached auxiliary components. The one or more second models may include one or more classification models trained to perform image classification. In some embodiments, the one or more classification models may include a plurality of classification models, each trained to perform classification for a different type of auxiliary component, such as a first model that identifies buttons, a second model that identifies attachments, a third model that identifies power ridges, and / or a fourth model that identifies brackets.

[0219] In the third approach, the auxiliary component detector 470 may combine 3D dentition model registration with per-tooth or per-auxiliary-component processing. The auxiliary component detector 470 may obtain a 3D dentition model 455 including a depiction of the patient's dentition, wherein the 3D dentition model 455 indicates one or more expected auxiliary components, and may register the 3D dentition model 455 to the 2D dentition image 448. The auxiliary component detector 470 may determine expected locations of the one or more expected auxiliary components based at least in part on information from the 3D dentition model 455. A cropped 2D image may be associated with a tooth and optionally an expected auxiliary component on the tooth, and the cropped 2D image may include pixels of the 2D dentition image 448 for the associated expected auxiliary component and optionally exclude at least some pixels of the 2D dentition image 448 for the tooth. The cropped 2D image may have a size that is larger than a size of the expected auxiliary component in the 2D dentition image 448 by up to a threshold number of pixels (e.g., 5 pixels, 10 pixels, 20 pixels, etc.). In some embodiments, the one or more cropped 2D images may include a first cropped 2D image associated with a tooth and a first expected auxiliary component on the tooth and a second cropped 2D image associated with the tooth and a second expected auxiliary component on the tooth. Accordingly, multiple cropped images may be generated for a single tooth in instances where a tooth has multiple expected auxiliary components. This third approach enables focused analysis on specific locations where auxiliary components are expected, reduces false positives, and enables multiple auxiliary components on a single tooth to be analyzed separately.

[0220] In some embodiments, the auxiliary component detector 470 may determine, based on a treatment plan, one or more types of expected auxiliary components for a tooth, and may determine which classification models of the plurality of classification models to apply to the cropped 2D image based on the one or more types of expected auxiliary components, wherein the cropped 2D image is processed using the determined classification models and is not processed by a remainder of the plurality of classification models. In some embodiments, an output of the plurality of classification models with reference to a cropped 2D image may include a plurality of confidence values, each associated with a different type of auxiliary component. The auxiliary component detector 470 mayAttorney Docket No.: 28510.983 (L0820PCT)determine a type of auxiliary component that is present at the tooth associated with the cropped 2D image based on determining that the confidence value associated with the type of auxiliary component satisfies one or more criteria. The one or more criteria may include a first criterion that a confidence value associated with the type of auxiliary component is greater than confidence values associated with a remainder of types of auxiliary components, and / or a second criterion that the confidence value associated with the type of auxiliary component exceeds a confidence threshold.

[0221] In some embodiments, the auxiliary component detector 470 may perform longitudinal monitoring of auxiliary components. The auxiliary component detector 470 may obtain a new 2D dentition image including a new depiction of the patient's dentition ata second time that is later than a first time at which the 2D dentition image 448 is obtained, process the new 2D dentition image using the one or more first models, generate one or more new cropped 2D images, process the one or more new cropped 2D images using the one or more second models, make a comparison of new indications generated from processing of the new cropped 2D images to indications generated from processing of the cropped 2D images, determine any differences between the new indications and the indications based on the comparison, and output a notice of the differences.

[0222] The 3D dentition model 455 may be a virtual model of an upper and / or lower dental arch of the patient. The 3D dentition model 455 may be a model that is associated with a current stage of dental treatment in embodiments. Accordingly, a shape, arrangement, etc. of the patient's teeth in the 3D dentition model 455 should approximately match the current shape, arrangement, etc. of the patient's teeth.

[0223] In an example, a dental professional may subject a patient to intraoral scanning prior to dental treatment. In doing so, the dental professional may apply an intraoral scanner to one or more patient intraoral locations. The scanning may be divided into one or more segments (also referred to as roles). As an example, the segments may include a lower dental arch of the patient, an upper dental arch of the patient, one or more preparation teeth of the patient (e.g., teeth of the patient to which a dental device such as a crown or other dental prosthetic will be applied), one or more teeth which are contacts of preparation teeth (e.g., teeth not themselves subject to a dental device but which are located next to one or more such teeth or which interface with one or more such teeth upon mouth closure), and / or patient bite (e.g., scanning performed with closure of the patient's mouth with the scan being directed towards an interface area of the patient's upper and lower teeth). Via such scanner application, the intraoral scanner may provide intraoral scan data, which may be 3D image data in the form of one or more points (e.g., one or more point clouds, height maps, etc.).

[0224] When a scan session or a portion of a scan session associated with a particular scanning role (e.g., upper jaw role, lower jaw role, bite role, etc.) is complete (e.g., all scans for an intraoral site orAttorney Docket No.: 28510.983 (L0820PCT)dental site have been captured), processing logic may generate a virtual 3D model (e.g., a 3D dentition model 455) of one or more scanned dental sites (e.g., of an upper jaw and a lower jaw). The final 3D model may be a set of 3D points and their connections with each other (i.e. a mesh). To generate the virtual 3D model, processing logic may register and stitch together the intraoral scans generated from the intraoral scan session that are associated with a particular scanning role. In one embodiment, performing scan registration includes capturing 3D data of various points of a surface in multiple scans, and registering the scans by computing transformations between the scans. The 3D data may be projected into a 3D space of a 3D model to form a portion of the 3D model. The intraoral scans may be integrated into a common reference frame by applying appropriate transformations to points of each registered scan and projecting each scan into the 3D space.

[0225] In one embodiment, registration is performed for adjacent or overlapping intraoral scans (e.g., each successive frame of an intraoral video). Processing logic may repeat registration for all intraoral scans of a sequence of intraoral scans to obtain transformations for each intraoral scan, to register each intraoral scan with previous intraoral scan(s) and / or with a common reference frame (e.g., with the 3D model). Processing logic may integrate intraoral scans into a single virtual 3D model by applying the appropriate determined transformations to each of the intraoral scans. Each transformation may include rotations about one to three axes and translations within one to three planes.

[0226] Once a 3D dentition model 455 is generated of a patient's current (pre-treatment) dentition, a dental treatment application (e.g., optionally running on dental treatment computing system 510) may process the 3D dentition model 455 to generate a treatment plan. A dental treatment plan for orthodontic treatment may include a treatment plan to apply a sequence of aligners to a patient's teeth to correct malocclusions, for example. The dental treatment plan may include a series of 3D dentition models 455, each associated with a different stage of dental treatment. These 3D dentition models 455 may be used to fabricate dental appliances such as orthodontic aligners, palatal expanders, retainers, and so on. Each dental appliance may correspond to a particular stage of dental treatment and may be unique to the patient for that stage of dental treatment. At a given stage of dental treatment, the dental appliance associated with that stage of dental treatment may be worn by the patient to effect the dental treatment (e.g., to apply forces to the patient's teeth and / or palate to reposition those teeth and / or widen the palate).

[0227] In some instances, the dental treatment plan calls for auxiliary components on one or more teeth of the patient. The auxiliary components may be similar to the auxiliary components 204 depicted in FIG.2A or FIG.2B. One type of auxiliary component, referred to as dental attachments, may facilitate application of one or more forces to one or more teeth, such as application of rotational forces to teeth that may be difficult to achieve without such attachments. Other types of auxiliaryAttorney Docket No.: 28510.983 (L0820PCT)components perform other functions that may have clinical significance. Accordingly, it can be important that the auxiliary components are applied to the patient's teeth as planned.

[0228] In one embodiment, auxiliary component detector 470 processes dentition images 448 and / or dentition models 455 to determine whether the patient has one or more missing auxiliary components. A missing auxiliary component may be identified by a trained ML model in some instances. In some embodiments, missing auxiliary components are identified by comparing patient data (e.g., an image of a patient's current dentition, such as a 2D dentition image 448) to data from the dental treatment plan (e.g., a 3D model 455 or point cloud of the planned dentition of the patient for the current stage of treatment or a projection of the 3D model 455 or point cloud onto a plane of one or more images of the image data). Based on the comparison, processing logic may identify regions where auxiliary components should be present but are absent. In some embodiments, processing logic may additionally make a recommendation to change one or more auxiliary components if one or more auxiliary component replacement criteria are satisfied. In some cases, processing logic may identify a damaged auxiliary component (e.g., an ML model may process an image of a patient's dentition, which may output an indication of a damaged auxiliary component). In such instances, processing logic may recommend replacing the auxiliary component that has been damaged.

[0229] In embodiments using the first approach, the auxiliary component detector 470 may register the 3D dentition model 455 for the current stage of treatment to one or more 2D dentition images 448 captured by patient device 520 and determine, using the registered 3D dentition model 455 and the 2D dentition image(s) 448, whether one or more expected auxiliary components are present in the 2D dentition image 448 (e.g., whether any auxiliary components are missing). In embodiments using the second approach, the auxiliary component detector 470 may segment the 2D dentition image 448 to identify individual teeth, generate cropped 2D images for each tooth, and process the cropped 2D images using one or more classification models to determine whether auxiliary components are presenton each tooth. In embodiments using the third approach, the auxiliary component detector 470 may use 3D dentition model registration to identify expected locations of auxiliary components and then generate cropped 2D images based on those expected locations for per-auxiliary-component classification. If one or more auxiliary components are missing from the 2D dentition image 448, the auxiliary component detector 470 may cause the Ul 532 to output an indication of the missing auxiliary component(s), may outputa recommendation forthe patient to make an in-person visit to the dental professional, may recommend replacing the missing dental auxiliary, and so on. For example, the dental treatment computing system 510 may be configured to transmit the indication to the patient device 520 and / or the dental professional device 530. The patient device 520 and / or the dentalAttorney Docket No.: 28510.983 (L0820PCT)professional device 530 may be configured to output the indication to a display (e.g., a display presenting the III 532).

[0230] Typically, auxiliary components to a patient's teeth are applied at a start of treatment. However, those auxiliary components may not be needed for all stages of treatment. For example, one or more stages of treatment may use the auxiliary components to apply specific forces to one or more teeth. After those stages of treatment are complete, then the auxiliary components may no longer have a clinical significance. Accordingly, in some embodiments processing logic determines whether any identified missing auxiliary components have clinical significance. Such a determination may be made by determining whether tooth rotations / motions facilitated by the missing auxiliary components have already been performed or are yet to be performed. The missing auxiliary component(s) may have clinical significance if one or more tooth movements achieved using forces facilitated by the missing auxiliary components(s) have not yet been performed. If there are no future movements / rotations of teeth that require the missing auxiliary components for the remainder of treatment, then the missing auxiliary components may not have clinical significance.

[0231] Processing logic may then perform one or more actions based on the identified missing auxiliary components and / or based on the determination of whether the missing auxiliary components have clinical significance. For example, if the missing auxiliary components have clinical significance, then a recommendation for the patient to have an in-person visit with the doctor to enable the doctor to reattach the missing auxiliary components may be output to the patient device and / or the doctor device. If the missing auxiliary components are determined not to have clinical significance, then no such recommendation may be output. In one embodiment, if a missing auxiliary component with clinical significance is identified, then treatment may be slowed down (e.g., the timing of advancing to a next treatment stage may be delayed). For example, a current treatment stage may be prolonged until the patient has a next scheduled in-patient doctor visit. In some embodiments, a notice of a missing auxiliary component and / or whether it has clinical significance may be output to a device of a doctor, enabling the doctor to make a decision on howto proceed.

[0232] In some embodiments, the auxiliary component detector 470 can include an auxiliary component detection model. The auxiliary component detection model may include an Al model that uses a 2D dentition image 448 as input and identifies one or more auxiliary components depicted in the 2D dentition image 448. Such an Al model can be one or more of decision trees (e.g., random forests), support vector machines, logistic regression, K-nearest neighbor (KNN), or other types of machine learning models, for example. In one embodiment, such an Al model can be or include one or more artificial neural networks (also referred to simply as a neural network). The artificial neural network can be, for example, a convolutional neural network (CNN) or a deep neural network. In one embodiment,Attorney Docket No.: 28510.983 (L0820PCT)processing logic performs supervised machine learning to train the neural network. In some embodiments, the auxiliary component detector 470 may include a tooth segmenter / cl assifier 435 for performing segmentation and classification operations on input data to identify individual teeth, and an image cropper 437 for generating cropped images based on the segmentation information or based on expected auxiliary component locations from a registered 3D dentition model.

[0233] In some embodiments, the dental treatment datastore 514 can be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes or hard drives, network-attached storage (NAS), storage area network (SAN), and so forth. In some embodiments, the datastore 514 can be a network-attached file server, while in other embodiments, the datastore 514 can be or can host some other type of persistent storage such as an object-oriented database, a relational database, and so forth. The datastore 514 can be hosted by any of the platforms or devices associated with the system 500 (e.g. the dental treatment computing system 510). In other embodiments, the datastore 514 can be on or hosted by one or more different machines coupled to the treatment coordination platform via the network 540. In some cases, the datastore 514 can store portions of image, video, or text data received from the client devices 520, 530 and / or any platform and any of its associated modules.

[0234] In one embodiment, the dentition images 515 may include 2D dentition images 448 depicting patients' dentitions. The 2D dentition images 448 may have been received from patient devices (e.g., the patient device 520) and may include images captured by a camera 522 of a patient device 520. A 2D dentition image 448 may include a 2D image of at least a portion of the patient's dentition and surrounding oral features (e.g., gums, tongue, etc.). Each 2D dentition image 448 of the dentition images 515 may include metadata identifying the patient whose dentition is depicted in the 2D dentition image 448, a timestamp indicating the time and / or date when the 2D dentition image 448 was captured, and / or other data. In some embodiments, the dentition models 516 may include 3D dentition models 455 depicting patients' dentitions. A 3D dentition model 455 may include a 3D model of at least a portion of a patient's dentition and surrounding oral features. The 3D model may include a computergenerated model generated from images captured by hardware used by an orthodontic professional. Each 3D dentition model 455 of the dentition models 516 may include metadata identifying the patient whose dentition is depicted in the 3D dentition model 455, a timestamp indicating the time and / or date when the 3D dentition model 455 was generated, and / or other data.

[0235] It is appreciated that in some implementations, the functions of the dental treatment computing system 510 can be provided by a fewer number of machines. For example, in some implementations, functionalities of the dental treatment computing system 510 can be integrated into a single machine, while in other implementations, functionalities of the dental treatment computingAttorney Docket No.: 28510.983 (L0820PCT)system 510 can be integrated into multiple, or more, machines. In addition, in some implementations, only some platforms of the system can be integrated into a combined platform.

[0236] While the subsystems and components of the dental treatment computing system 510 are described separately, it should be understood that the functionalities can be divided differently or integrated in various ways within the platform while still applying similar functionality for the system. Furthermore, the dental treatment computing system 510 and the associated subsystems and components can be implemented in various forms, such as standalone applications, web-based platforms, integrated systems within larger software suites, or dedicated hardware devices, just to name a few possible forms.

[0237] In general, certain functions described in embodiments as being performed by the dental treatment computing system 510 can also be performed by client devices 520, 530. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together. The dental treatment computing system 510 can also be accessed as a service provided to other systems or devices through appropriate application programming interfaces, and thus is not limited to use in websites.

[0238] It is appreciated that in some implementations, the dental treatment computing system 510 or client devices 520, 530 of the system 500 can each include an associated API, or mechanism for communicating with APIs. In such a way, any of the components of system 500 can support instructions and / or communication mechanisms that can be used to communicate data requests and formats of data to and from any other component of system 500, in addition to communicating with APIs external to the system (e.g., not shown in FIG. 5).

[0239] In some embodiments of the disclosure, a "user" can be represented as a single individual. However, other implementations of the disclosure encompass a "user" being an entity controlled by a set of users and / or an automated source. For example, a set of individual users federated as a community in a social network can be considered a "user." In another example, an automated consumer can be an automated ingestion pipeline, such as a topic channel.

[0240] In situations in which the systems, or components therein, discussed here collect personal information about users, or can make use of personal information, the users can be provided with an opportunity to control whether the system or components collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and / or how to receive content from the system or components that can be more relevant to the user. In addition, certain data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity can be treated so that no personally identifiable information can be determined for the user, or aAttorney Docket No.: 28510.983 (L0820PCT)user's geographic location can be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user can have control over how information is collected about the user and used by the system and components.

[0241] FIG. 6 illustrates a block diagram of an example processing device 600 operating in accordance with one or more aspects of the present disclosure. In one implementation, the processing device 600 can be a part of any computing device of FIG. 1, or any combination thereof. Example processing device 600 can be connected to other processing devices in a LAN, an intranet, an extranet, and / or the Internet. The processing device 600 can be a personal computer (PC), a set-top box (STB), a server, a network router, switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single example processing device is illustrated, the term “processing device” shall also be taken to include any collection of processing devices (e.g., computers) that individually or jointly execute a set (or multiple sets) of instructions to perform anyone or more of the methods discussed herein.

[0242] Example processing device 600 can include a processor 602 (e.g., a CPU), a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 618), which can communicate with each other via a bus 630.

[0243] Processor 602 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, processor 602 can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processor 602 can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. In accordance with one or more aspects of the present disclosure, processor 602 can be configured to execute instructions (e.g. processing logic 626 can implement the auxiliary component detector 470, the training data labeler 434, or other components discussed herein).

[0244] Example processing device 600 can further include a network interface device 608, which can be communicatively coupled to a network 620. Example processing device 600 can further comprise a video display 610 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 612 (e.g., a keyboard), an input control device 614 (e.g., a cursorAttorney Docket No.: 28510.983 (L0820PCT)control device, a touch-screen control device, a mouse), and a signal generation device 616 (e.g., an acoustic speaker).

[0245] Data storage device 618 can include a computer-readable storage medium (or, more specifically, a non-transitory computer-readable storage medium) 628 on which is stored one or more sets of executable instructions 622. In accordance with one or more aspects of the present disclosure, executable instructions 622 can comprise executable instructions (e.g. instructions for implementing the auxiliary component detector 470, the training data labeler 434, or other components discussed herein).

[0246] Executable instructions 622 can also reside, completely or at least partially, within main memory 604 and / or within processor 602 during execution thereof by example processing device 600, main memory 604 and processor 602 also constituting computer-readable storage media. Executable instructions 622 can further be transmitted or received over a network (e.g., the network 540) via network interface device 608.

[0247] While the computer-readable storage medium 628 is shown in FIG. 6 as a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more sets of operating instructions. The term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine that cause the machine to perform any one or more of the methods described herein. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.

[0248] FIG. 7A illustrates an exemplary tooth repositioning dental appliance or aligner 700 that can be worn by a patient in order to achieve an incremental repositioning of individual teeth 702 in the jaw. The aligner 700 may be formed using modular trays and / or global plan data, as disclosed herein. The appliance can include a shell (e.g., a continuous polymeric shell or a segmented shell) having teeth-receiving cavities that receive and resiliently reposition the teeth. An appliance or portion(s) thereof may be indirectly fabricated using a physical model of teeth. For example, an appliance (e.g., polymeric appliance) can be formed using a physical model of teeth and a sheet of suitable layers of polymeric material. A “polymeric material,” as used herein, may include any material formed from a polymer. A “polymer,” as used herein, may refer to a molecule composed of repeating structural units connected by covalent chemical bonds often characterized by a substantial number of repeating units (e.g., equal to or greater than 3 repeating units, optionally, in some embodiments equal to or greater than 10 repeating units, in some embodiments greater or equal to 30 repeating units) and a high molecular weight (e.g. greater than or equal to 10,000 Da, in some embodiments greater than or equal to 50,000 Da or greater than or equal to 100,000 Da).Attorney Docket No.: 28510.983 (L0820PCT)

[0249] Although polymeric aligners are discussed herein, the techniques disclosed may also be applied to aligners having different materials. Some embodiments are discussed herein with reference to orthodontic aligners (also referred to simply as aligners). However, embodiments also extend to other types of shells formed over molds, such as orthodontic retainers, orthodontic splints, sleep appliances for mouth insertion (e.g., for minimizing snoring, sleep apnea, etc.), palatal expanders and / or shells for non-dental applications. Accordingly, it should be understood that embodiments herein that refer to aligners also apply to other types of shells.

[0250] The aligner 700 can fit over all teeth present in an upper or lower jaw, or less than all of the teeth. The appliance can be designed specifically to accommodate the teeth of the patient (e.g., the topography of the tooth-receiving cavities matches the topography of the patients teeth) and may be fabricated based on positive or negative models of the patients teeth generated by impression, scanning, and the like. Alternatively, the appliance can be a generic appliance configured to receive the teeth, but not necessarily shaped to match the topography of the patients teeth. In some cases, only certain teeth received by an appliance will be repositioned by the appliance while other teeth can provide a base or anchor region for holding the appliance in place as it applies force against the tooth or teeth targeted for repositioning. In some cases, some, most, or even all of the teeth will be repositioned at some pointduring treatment. Teeth that are moved can also serve as a base or anchor for holding the appliance as it is worn by the patient. Typically, no wires or other means will be provided for holding an appliance in place over the teeth. In some cases, however, it may be desirable or necessary to provide individual dental auxiliaries (e.g., dental attachments or other anchoring elements) 704 on teeth 702 with corresponding receptacles or apertures 706 in the aligner 700 so that the appliance can apply a selected force on the tooth. Exemplary appliances, including those utilized in the Invisalign® System, are described in numerous patents and patent applications assigned to Align Technology, Inc. including, for example, in U.S. Patent Nos. 6,450,807, and 5,975,893, as well as on the company’s website, which is accessible on the World Wide Web (see, e.g., the URL “invisalign.com”). Examples of tooth-mounted dental auxiliaries suitable for use with orthodontic appliances are also described in patents and patent applications assigned to Align Technology, Inc., including, for example, U.S. Patent Nos. 6,309,215 and 6,830,450.

[0251] FIG. 7B illustrates a tooth repositioning system 710 including a plurality of appliances 712, 714, 716. Any of the appliances described herein can be designed and / or provided as part of a set of a plurality of appliances used in a tooth repositioning system. Each appliance may be configured so a tooth-receiving cavity has a geometry corresponding to an intermediate or final tooth arrangement intended for the appliance. The patients teeth can be progressively repositioned from an initial tooth arrangement to a target tooth arrangement by placing a series of incremental position adjustmentAttorney Docket No.: 28510.983 (L0820PCT)appliances over the patients teeth. For example, the tooth repositioning system 710 can include a first appliance 712 corresponding to an initial tooth arrangement, one or more intermediate appliances 714 corresponding to one or more intermediate arrangements, and a final appliance 716 corresponding to a target arrangement. A target tooth arrangement can be a planned final tooth arrangement selected for the patients teeth at the end of all planned orthodontic treatment. Alternatively, a target arrangement can be one of some intermediate arrangements for the patients teeth during the course of orthodontic treatment, which may include various different treatment scenarios, including, but not limited to, instances where surgery is recommended, where interproximal reduction (IPR) is appropriate, where a progress check is scheduled, where anchor placement is best, where palatal expansion is desirable, where restorative dentistry is involved (e.g., inlays, onlays, crowns, bridges, implants, veneers, and the like), etc. As such, it is understood that a target tooth arrangement can be any planned resulting arrangement for the patients teeth that follows one or more incremental repositioning stages. Likewise, an initial tooth arrangement can be any initial arrangement for the patient's teeth that is followed by one or more incremental repositioning stages.

[0252] In some embodiments, the appliances 712, 714, 716 (or portions thereof) can be produced using indirect fabrication techniques, such as by thermoforming over a positive or negative mold. Indirect fabrication of an orthodontic appliance can involve producing a positive or negative mold of the patients dentition in a target arrangement (e.g., by rapid prototyping, milling, etc.) and thermoforming one or more sheets of material over the mold in order to generate an appliance shell.

[0253] In an example of indirect fabrication, a mold of a patients dental arch may be fabricated from a digital model of the dental arch, and a shell may be formed over the mold (e.g., by thermoforming a polymeric sheet over the mold of the dental arch and then trimming the thermoformed polymeric sheet). The fabrication of the mold may be performed by a rapid prototyping machine (e.g., a stereolithography (SLA) 3D printer). The rapid prototyping machine may receive digital models of molds of dental arches and / or digital models of the appliances 712, 714, 716 after the digital models of the appliances 712, 714, 716 have been processed by processing logic of a computing device, such as the computing device in FIG. 6. The processing logic may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed by a processing device), firmware, or a combination thereof.

[0254] To manufacture the molds, a shape of a dental arch for a patient at a treatment stage is determined based on a treatment plan. In the example of orthodontics, the treatment plan may be generated based on an intraoral scan of a dental arch to be modeled. The intraoral scan of the patients dental arch may be performed to generate a 3D virtual model of the patients dental arch (mold). For example, a full scan of the mandibular and / or maxillary arches of a patient may be performed toAttorney Docket No.: 28510.983 (L0820PCT)generate 3D virtual models thereof. The intraoral scan may be performed by creating multiple overlapping intraoral images from different scanning stations and then stitching together the intraoral images to provide a composite 3D virtual model. In other applications, virtual 3D models may also be generated based on scans of an object to be modeled or based on use of computer aided drafting techniques (e.g., to design the virtual 3D mold). Alternatively, an initial negative mold may be generated from an actual object to be modeled (e.g., a dental impression or the like). The negative mold may then be scanned to determine a shape of a positive mold that will be produced.

[0255] Once the virtual 3D model of the patients dental arch is generated, a dental practitioner and / or treatment planning software may determine a desired treatment outcome, which includes final positions and orientations for the patients teeth. Processing logic may then determine a number of treatment stages to cause the teeth to progress from starting positions and orientations to the target final positions and orientations. The shape of the final virtual 3D model and each intermediate virtual 3D model may be determined by computing the progression of tooth movement throughout orthodontic treatment from initial tooth placement and orientation to final corrected tooth placement and orientation. For each treatment stage, a separate virtual 3D model of the patients dental arch at that treatment stage may be generated. The shape of each virtual 3D model will be different. The original virtual 3D model, the final virtual 3D model and each intermediate virtual 3D model is unique and customized to the patient.

[0256] Accordingly, multiple different virtual 3D models (digital designs) of a dental arch may be generated for a single patient. A first virtual 3D model may be a unique model of a patients dental arch and / or teeth as they presently exist, and a final virtual 3D model may be a model of the patients dental arch and / or teeth after correction of one or more teeth and / or a jaw. Multiple intermediate virtual 3D models may be modeled, each of which may be incrementally different from previous virtual 3D models.

[0257] Each virtual 3D model of a patients dental arch may be used to generate a unique customized physical mold of the dental arch at a particular stage of treatment. The shape of the mold may be at least in part based on the shape of the virtual 3D model for that treatment stage. The virtual 3D model may be represented in a file such as a computer aided drafting (CAD) file or a 3D printable file such as a stereolithography (STL) file. The virtual 3D model for the mold may be sent to a third party (e.g., clinician office, laboratory, manufacturing facility or other entity). The virtual 3D model may include instructions that will control a fabrication system or device in order to produce the mold with specified geometries.

[0258] A clinician office, laboratory, manufacturing facility or other entity may receive the virtual 3D model of the mold, the digital model having been created as set forth above. The entity may input the digital model into a rapid prototyping machine. The rapid prototyping machine then manufacturesAttorney Docket No.: 28510.983 (L0820PCT)the mold using the digital model. One example of a rapid prototyping manufacturing machine is a 3D printer. 3D printing includes any layer-based additive manufacturing processes. 3D printing may be achieved using an additive process, where successive layers of material are formed in proscribed shapes. 3D printing may be performed using extrusion deposition, granular materials binding, lamination, photopolymerization, continuous liquid interface production (CLIP), or other techniques. 3D printing may also be achieved using a subtractive process, such as milling.

[0259] Appliances may be formed from each mold and when applied to the teeth of the patient, may provide forces to move the patients teeth as dictated by the treatment plan. The shape of each appliance is unique and customized for a particular patient and a particular treatment stage. In an example, the appliances 712, 714, 716 can be pressure formed or thermoformed over the molds. Each mold may be used to fabricate an appliance that will apply forces to the patients teeth at a particular stage of the orthodontic treatment. The appliances 712, 714, 716 each have teeth-receiving cavities that receive and resiliency reposition the teeth in accordance with a particular treatment stage.

[0260] In one embodiment, a sheet of material is pressure formed or thermoformed over the mold. The sheet may be, for example, a sheet of polymeric (e.g., an elastic thermopolymeric, a sheet of polymeric material, etc.). To thermoform the shell over the mold, the sheet of material may be heated to a temperature at which the sheet becomes pliable. Pressure may concurrently be applied to the sheet to form the now pliable sheet around the mold. Once the sheet cools, it will have a shape that conforms to the mold. In one embodiment, a release agent (e.g., a non-stick material) is applied to the mold before forming the shell. This may facilitate later removal of the mold from the shell. Forces may be applied to lift the appliance from the mold. In some instances, a breakage, warpage, or deformation may result from the removal forces. Accordingly, embodiments disclosed herein may determine where the probable pointor points of damage may occur in a digital design of the appliance prior to manufacturing and may perform a corrective action.

[0261] After an appliance is formed over a mold for a treatment stage, the appliance is removed from the mold (e.g., automated removal of the appliance from the mold), and the appliance is subsequently trimmed along a cutline (also referred to as a trim line). The processing logic may determine a cutline for the appliance. The determination of the cutline(s) may be made based on the virtual 3D model of the dental arch ata particular treatment stage, based on a virtual 3D model of the appliance to be formed over the dental arch, or a combination of a virtual 3D model of the dental arch and a virtual 3D model of the appliance. The location and shape of the cutline can be important to the functionality of the appliance (e.g., an ability of the appliance to apply desired forces to a patients teeth) as well as the fit and comfort of the appliance. For shells such as orthodontic appliances, orthodontic retainers and orthodontic splints, the trimming of the shell may play a role in the efficacy of the shell forAttorney Docket No.: 28510.983 (L0820PCT)its intended purpose (e.g., aligning, retaining or positioning one or more teeth of a patient) as well as the fit of the shell on a patients dental arch. For example, if too much of the shell is trimmed, then the shell may lose rigidity and an ability of the shell to exert force on a patients teeth may be compromised. When too much of the shell is trimmed, the shell may become weaker at that location and may be a point of damage when a patient removes the shell from their teeth or when the shell is removed from the mold. In some embodiments, the cut line may be modified in the digital design of the appliance as one of the corrective actions taken when a probable point of damage is determined to exist in the digital design of the appliance.

[0262] On the other hand, if too little of the shell is trimmed, then portions of the shell may impinge on a patients gums and cause discomfort, swelling, and / or other dental issues. Additionally, if too little of the shell is trimmed at a location, then the shell may be too rigid at that location. In some embodiments, the cutline may be a straight line across the appliance at the gingival line, below the gingival line, or above the gingival line. In some embodiments, the cutline may be a gingival cutline that represents an interface between an appliance and a patients gingiva. In such embodiments, the cutline controls a distance between an edge of the appliance and a gum line or gingival surface of a patient.

[0263] Each patient has a unique dental arch with unique gingiva. Accordingly, the shape and position of the cutline may be unique and customized for each patient and for each stage of treatment. For instance, the cutline is customized to follow along the gum line (also referred to as the gingival line). In some embodiments, the cutline may be away from the gum line in some regions and on the gum line in other regions. For example, it may be desirable in some instances for the cutline to be away from the gum line (e.g., not touching the gum) where the shell will touch a tooth and on the gum line (e.g., touching the gum) in the interproximal regions between teeth. Accordingly, it is important that the shell be trimmed along a predetermined cutline.

[0264] In some embodiments, the dental appliances (e.g., orthodontic appliances) herein (or portions thereof) can be produced using direct fabrication, such as additive manufacturing techniques (also referred to herein as “3D printing) or subtractive manufacturing techniques (e.g., milling). In some embodiments, direct fabrication involves forming an object (e.g., an orthodontic appliance or a portion thereof) without using a physical template (e.g., mold, mask etc.) to define the object geometry. Additive manufacturing techniques can be categorized as follows: (1) vat photopolymerization (e.g., stereolithography), in which an object is constructed layer by layer from a vat of liquid photopolymer resin; (2) material jetting, in which material is jetted onto a build platform using either a continuous or drop on demand (DOD) approach; (3) binder jetting, in which alternating layers of a build material (e.g., a powder-based material) and a binding material (e.g., a liquid binder) are deposited by a print head; (4) fused deposition modeling (FDM), in which material is drawn though a nozzle, heated, and depositedAttorney Docket No.: 28510.983 (L0820PCT)layer by layer; (5) powder bed fusion, including but not limited to direct metal laser sintering (DMLS), electron beam melting (EBM), selective heat sintering (SHS), selective laser melting (SLM), and selective laser sintering (SLS); (6) sheet lamination, including but not limited to laminated object manufacturing (LOM) and ultrasonic additive manufacturing (UAM); and (7) directed energy deposition, including but not limited to laser engineering net shaping, directed light fabrication, direct metal deposition, and 3D laser cladding. For example, stereolithography can be used to directly fabricate one or more of the appliances 712, 714, and 716. In some embodiments, stereolithography involves selective polymerization of a photosensitive resin (e.g., a photopolymer) according to a desired cross-sectional shape using light (e.g., ultraviolet light). The object geometry can be built up in a layer-bylayerfashion by sequentially polymerizing a plurality of object cross-sections. As another example, the appliances 712, 714, and 716 can be directly fabricated using selective laser sintering. In some embodiments, selective laser sintering involves using a laser beam to selectively melt and fuse a layer of powdered material according to a desired cross-sectional shape in order to build up the object geometry. As yet another example, the appliances 712, 714, and 716 can be directly fabricated by fused deposition modeling. In some embodiments, fused deposition modeling involves melting and selectively depositing a thin filament of thermoplastic polymer in a layer-by-layer manner in order to form an object. In yet another example, material jetting can be used to directly fabricate the appliances 712, 714, and 716. In some embodiments, material jetting involves jetting or extruding one or more materials onto a build surface in order to form successive layers of the object geometry.

[0265] FIG.7C illustrates a method 750 of orthodontic treatment using a plurality of appliances, in accordance with embodiments. One or more of the plurality ofappliances may be formed using modular trays and / or global plan data, as disclosed herein. The method 750 can be practiced using any of the appliances or appliance sets described herein. In block 760, a first orthodontic appliance is applied to a patients teeth in orderto reposition the teeth from a first tooth arrangement to a second tooth arrangement. In block 770, a second orthodontic appliance is applied to the patients teeth in orderto reposition the teeth from the second tooth arrangement to a third tooth arrangement. The method 750 can be repeated as necessary using any suitable number and combination of sequential appliances in order to incrementally reposition the patients teeth from an initial arrangement to a target arrangement. The appliances can be generated all at the same stage or in sets or batches (e.g., at the beginning of a stage of the treatment), or the appliances can be fabricated one at a time, and the patient can wear each appliance until the pressure of each appliance on the teeth can no longer be felt or until the maximum amount of expressed tooth movement forthat given stage has been achieved. A plurality of different appliances (e.g., a set) can be designed and even fabricated prior to the patient wearing any appliance of the plurality. After wearing an appliance for an appropriate period of time, the patient canAttorney Docket No.: 28510.983 (L0820PCT)replace the current appliance with the subsequent appliance in the series until no more appliances remain. The appliances are generally not affixed to the teeth and the patient may place and replace the appliances at any time during the procedure (e.g., patient-removable appliances). The final appliance or several appliances in the series may have a geometry or geometries selected to overcorrect the tooth arrangement. For instance, one or more appliances may have a geometry that would (if fully achieved) move individual teeth beyond the tooth arrangement that has been selected as the "final." Such overcorrection may be desirable in order to offset potential relapse after the repositioning method has been terminated (e.g., permit movement of individual teeth back toward their pre-corrected positions). Overcorrection may also be beneficial to speed the rate of correction (e.g., an appliance with a geometry that is positioned beyond a desired intermediate or final position may shift the individual teeth toward the position ata greater rate). In such cases, the use of an appliance can be terminated before the teeth reach the positions defined by the appliance. Furthermore, over-correction may be deliberately applied in order to compensate for any inaccuracies or limitations of the appliance.

[0266] FIG. 8 illustrates a method 800 for designing an orthodontic appliance to be produced by direct fabrication, in accordance with embodiments. Some or all of the blocks of the method 800 can be performed by any suitable data processing system or device, e.g., one or more processors configured with suitable instructions.

[0267] In block 810, a movement path to move one or more teeth from an initial arrangement to a target arrangement is determined. The initial arrangement can be determined from a mold ora scan of the patient's teeth or mouth tissue, e.g., using wax bites, direct contact scanning, x-ray imaging, tomographic imaging, sonographic imaging, and other techniques for obtaining information about the position and structure of the teeth, jaws, gums and other orthodontically relevant tissue. From the obtained data, a digital data set can be derived that represents the initial (e.g., pretreatment) arrangement of the patient's teeth and other tissues. Optionally, the initial digital data set is processed to segment the tissue constituents from each other. For example, data structures that digitally represent individual tooth crowns can be produced. Advantageously, digital models of entire teeth can be produced, including measured or extrapolated hidden surfaces and root structures, as well as surrounding bone and soft tissue.

[0268] The target arrangement of the teeth (e.g., a desired and intended end result of orthodontic treatment) can be received from a clinician in the form of a prescription, can be calculated from basic orthodontic principles, and / or can be extrapolated computationally from a clinical prescription. With a specification of the desired final positions of the teeth and a digital representation of the teeth themselves, the final position and surface geometry of each tooth can be specified to form a complete model of the tooth arrangement at the desired end of treatment.Attorney Docket No.: 28510.983 (L0820PCT)

[0269] Having both an initial position and a target position for each tooth, a movement path can be defined for the motion of each tooth. In some embodiments, the movement paths are configured to move the teeth in the quickest fashion with the least amount of round-tripping to bring the teeth from their initial positions to their desired target positions. The tooth paths can optionally be segmented, and the segments can be calculated so that each tooth's motion within a segment stays within threshold limits of linear and rotational translation. In this way, the end points of each path segment can constitute a clinically viable repositioning, and the aggregate of segment end points can constitute a clinically viable sequence of tooth positions, so that moving from one point to the next in the sequence does not result in a collision of teeth.

[0270] In block 820, a force system to produce movement of the one or more teeth along the movement path may be determined. A force system can include one or more forces and / or one or more torques. Different force systems can result in different types of tooth movement, such as tipping, translation, rotation, extrusion, intrusion, root movement, etc. Biomechanical principles, modeling techniques, force calculation / measurement techniques, and the like, including knowledge and approaches commonly used in orthodontia, may be used to determine the appropriate force system to be applied to the tooth to accomplish the tooth movement. In determining the force system to be applied, sources may be considered including literature, force systems determined by experimentation or virtual modeling, computer-based modeling, clinical experience, minimization of unwanted forces, etc.

[0271] The determination of the force system can include constraints on the allowable forces, such as allowable directions and magnitudes, as well as desired motions to be brought about by the applied forces. For example, in fabricating palatal expanders, different movement strategies may be desired for different patients. For example, the amount offeree needed to separate the palate can depend on the age of the patient, as very young patients may not have a fully formed suture. Thus, in juvenile patients and others without fully closed palatal sutures, palatal expansion can be accomplished with lower force magnitudes. Slower palatal movement can also aid in growing bone to fill the expanding suture. For other patients, a more rapid expansion may be desired, which can be achieved by applying larger forces. These requirements can be incorporated as needed to choose the structure and materials of appliances; for example, by choosing palatal expanders capable of applying large forces for rupturing the palatal suture and / or causing rapid expansion of the palate. Subsequent appliance stages can be designed to apply different amounts of force, such as first applying a large force to break the suture, and then applying smaller forces to keep the suture separated or gradually expand the palate and / or arch.Attorney Docket No.: 28510.983 (L0820PCT)

[0272] The determination of the force system can also include modeling of the facial structure of the patient, such as the skeletal structure of the jaw and palate. Scan data of the palate and arch, such as X-ray data or 3D optical scanning data, for example, can be used to determine parameters of the skeletal and muscular system of the patients mouth, so as to determine forces sufficient to provide a desired expansion of the palate and / or arch. In some embodiments, the thickness and / or density of the mid-palatal suture may be measured, or inputby a treating professional. In other embodiments, the treating professional can select an appropriate treatment based on physiological characteristics of the patient. For example, the properties of the palate may also be estimated based on factors such as the patients age— for example, young juvenile patients will typically require lower forces to expand the suture than older patients, as the suture has not yet fully formed.

[0273] In block 830, appliance design for an orthodontic appliance configured to produce the force system may be determined. Determination of the orthodontic appliance, appliance geometry, material composition, and / or properties can be performed using a treatment or force application simulation environment and / or application of machine learning. A simulation environment can include, e.g., computer modeling systems, biomechanical systems or apparatus, and the like. Optionally, digital models of the appliance and / or teeth can be produced, such as finite element models. The finite element models can be created using computer program application software available from a variety of vendors. For creating solid geometry models, computer aided engineering (CAE) or computer aided design (CAD) programs can be used, such as the AutoCAD® software products available from Autodesk, Inc., of San Rafael, CA. For creating finite element models and analyzing them, program products from a number of vendors can be used, including finite element analysis packages from ANSYS, Inc., of Canonsburg, PA, and SIMULIA(Abaqus) software products from Dassault Systemes of Waltham, MA.

[0274] Optionally, one or more orthodontic appliances can be selected fortesting or force modeling. As noted above, a desired tooth movement, as well as a force system required or desired for eliciting the desired tooth movement, can be identified. Using the simulation environment, a candidate orthodontic appliance can be analyzed or modeled for determination of an actual force system resulting from use of the candidate appliance. One or more modifications can optionally be made to a candidate appliance, and force modeling can be further analyzed as described, e.g., in order to iteratively determine an appliance design that produces the desired force system.

[0275] In block 840, instructions for fabrication of the orthodontic appliance incorporating the appliance design are generated. The instructions can be configured to control a fabrication system or device in order to produce the orthodontic appliance with the specified orthodontic appliance. In some embodiments, the instructions are configured for manufacturing the orthodontic appliance using directAttorney Docket No.: 28510.983 (L0820PCT)fabrication (e.g., stereolithography, selective laser sintering, fused deposition modeling, 3D printing, continuous direct fabrication, multi-material direct fabrication, etc.), in accordance with the various methods presented herein. In alternative embodiments, the instructions can be configured for indirect fabrication of the appliance, e.g., by thermoforming. In some embodiments, the instructions for fabrication of the orthodontic appliance include instructions for performing modular trays and / or global plan data, as disclosed herein.

[0276] Method 800 may comprise additional blocks: 1) The upper arch and palate of the patient is scanned intraorally to generate three-dimensional data of the palate and upper arch; and / or 2) The three-dimensional shape profile of the appliance is determined to provide a gap and teeth engagement structures.

[0277] Although the above blocks show a method 800 of designing an orthodontic appliance in accordance with some embodiments, a person of ordinary skill in the art will recognize some variations based on the teaching described herein. Some of the blocks may comprise sub-blocks. Some of the blocks may be repeated as often as desired. One or more blocks of the method 800 may be performed with any suitable fabrication system or device, such as the embodiments described herein. Some of the blocks may be optional, and the order of the blocks can be varied as desired.

[0278] FIG.9 illustrates a method 900 for digitally planning an orthodontic treatment and / or design or fabrication of an appliance, in accordance with embodiments. The method 900 can be applied to any of the treatment procedures described herein and can be performed by any suitable data processing system.

[0279] In block 910, a digital representation of a patients teeth is received. The digital representation can include surface topography data for the patients intraoral cavity (including teeth, gingival tissues, etc.). The surface topography data can be generated by directly scanning the intraoral cavity, a physical model (positive or negative) of the intraoral cavity, or an impression of the intraoral cavity, using a suitable scanning device (e.g., a handheld scanner, desktop scanner, etc.).

[0280] In block 920, one or more treatment stages are generated based on the digital representation of the teeth. The treatment stages can be incremental repositioning stages of an orthodontic treatment procedure designed to move one or more of the patients teeth from an initial tooth arrangement to a target arrangement. For example, the treatment stages can be generated by determining the initial tooth arrangement indicated by the digital representation, determining a target tooth arrangement, and determining movement paths of one or more teeth in the initial arrangement necessary to achieve the target tooth arrangement. The movement path can be optimized based on minimizing the total distance moved, preventing collisions between teeth, avoiding tooth movements that are more difficult to achieve, or any other suitable criteria.Attorney Docket No.: 28510.983 (L0820PCT)

[0281] In block 930, at least one orthodontic appliance is fabricated based on the generated treatment stages. For example, a set of appliances can be fabricated, each shaped according to a tooth arrangement specified by one of the treatment stages, such that the appliances can be sequentially worn by the patient to incrementally reposition the teeth from the initial arrangement to the target arrangement. The appliance set may include one or more of the orthodontic appliances described herein. The fabrication of the appliance may involve creating a digital model of the appliance to be used as input to a computer-controlled fabrication system. The appliance can be formed using direct fabrication methods, indirect fabrication methods, or combinations thereof, as desired. The fabrication of the appliance may include modular trays and / or global plan data, as disclosed herein.

[0282] In some instances, staging of various arrangements or treatment stages may not be necessary for design and / or fabrication of an appliance. As illustrated by the dashed line in FIG.9, design and / or fabrication of an orthodontic appliance, and perhaps a particular orthodontic treatment, may include use of a representation of the patients teeth (e.g., receive a digital representation of the patients teeth at block 910), followed by design and / or fabrication of an orthodontic appliance based on a representation of the patients teeth in the arrangement represented by the received representation.

[0283] Some examples have been described with reference to orthodontic treatment plans that include a series of stages that are each associated with a different orthodontic aligner. It should be understood that any such examples described with reference to orthodontic treatment and a series of orthodontic aligners also applies to palatal expansion treatment and a series of palatal expanders. For palatal expansion treatment, a similar process may be performed as described above for orthodontic treatment. For example, an upper and / or lower dental arch and upper palate may be scanned using an intraoral scanner to generate a 3D model of the dental arch(es) and of the upper palate. A final shape (e.g., width) of the upper palate may be determined, and a series of treatment stages to progress from a current upper palate shape and a final target upper palate shape may be determined. For each treatment stage, a polymeric palatal expander may be fabricated, either via direct fabrication (e.g., direct 3D printing) or by 3D printing of a mold and thermoforming a palatal expander over the mold. Materials used for palatal expanders may be the same as or different from those used for orthodontic aligners in embodiments.

[0284] Any of the methods (including user interfaces) described herein can be implemented as software, hardware or firmware, and can be described as a non-transitory machine-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like. For example,Attorney Docket No.: 28510.983 (L0820PCT)computer models (e.g., for additive manufacturing) and instructions related to forming a dental device can be stored on a non-transitory machine-readable storage medium.

[0285] It should be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiment exampleswill be apparent to those of skill in the art upon reading and understanding the above description. Although the present disclosure describes specific examples, it will be recognized that the systems and methods of the present disclosure are not limited to the examples described herein but can be practiced with modifications within the scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the present disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0286] The embodiments of methods, hardware, software, firmware, or code set forth above can be implemented via instructions or code stored on a machine-accessible, machine readable, computer accessible, or computer readable medium which are executable by a processing element. “Memory” includes any mechanism that provides (i.e., stores and / or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, “memory” includes random-access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM); ROM; magnetic or optical storage medium; flash memory devices; electrical storage devices; optical storage devices; acoustical storage devices, and any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0287] Reference throughout this specification to “one embodiment’ or “an embodiment’ means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Thus, the appearances ofthe phrases “in one embodiment’ or “in an embodiment’ in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0288] In the foregoing specification, a detailed description has been given with reference to specific exemplary embodiments. It will, however, be evident that various modifications and changes can be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. Furthermore, the foregoing use of embodiment, embodiment, and / or other exemplarily language does not necessarily refer to the same embodiment or the same example, but can refer to different and distinct embodiments, as well as potentially the same embodiment.Attorney Docket No.: 28510.983 (L0820PCT)

[0289] The words “example” or “exemplary” are used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example’ or “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from context, “X includes A or B” is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Moreover, use of the term “an embodiment’ or “one embodiment’ or “an embodiment’ or “one embodiment’ throughout is not intended to mean the same embodiment or embodiment unless described as such. Also, the terms “first,” “second,” “third,” “fourth,” etc. as used herein are meant as labels to distinguish among different elements and can not necessarily have an ordinal meaning according to their numerical designation.

[0290] A digital computer program, which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a digital computing environment. The essential elements of a digital computer a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and digital data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry or quantum simulators. Generally, a digital computer will also include-orbe operatively coupled to receive digital data from or transfer digital data to, or both-one or more mass storage devices for storing digital data, e.g., magnetic, magneto-optical disks, optical disks, or systems suitable for storing information. However, a digital computer need not have such devices.

[0291] Digital computer-readable media suitable for storing digital computer program instructions and digital data include all forms of non-volatile digital memory, media, and memory devices, including byway of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; CD-ROM and DVD-ROM disks.

[0292] Control of the various systems described in this specification, or portions of them, can be implemented in a digital computer program product that includes instructions that are stored on one orAttorney Docket No.: 28510.983 (L0820PCT)more non-transitory machine-readable storage media, and that are executable on one or more digital processing devices. The systems described in this specification, or portions of them, can each be implemented as an apparatus, method, or system that can include one or more digital processing devices and memory to store executable instructions to perform the operations described in this specification.

[0293] While this specification contains many specific embodiment details, these should not be construed as limitations on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a sub-combination or variation of a subcombination.

[0294] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software productor packaged into multiple software products.

[0295] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous. Certain example embodiments are now described.

[0296] Example embodiment 1 is a method comprising obtaining a two-dimensional (2D) dentition image comprising a depiction of a patient's dentition; obtaining a three-dimensional (3D) dentition model comprising a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliary components are present in the 2D dentitionAttorney Docket No.: 28510.983 (L0820PCT)image based at least in part on information from the 3D dentition model; and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

[0297] Example embodiment 2 may further extend example embodiment 1. In example embodiment 2, the 2D dentition image further comprises one or more auxiliary components; and determining whether the one or more expected auxiliary components are present in the 2D dentition image comprises matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image.

[0298] Example embodiment 3 may further extend any of example embodiments 1 -2. In example embodiment 3, the 3D dentition model indicates the one or more expected auxiliary components on one or more teeth of the patient at one or more target locations; and the 2D dentition image indicates the one or more auxiliary components atone or more locations in the 2D dentition image; and matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image comprises comparing the one or more locations of the respective auxiliary components in the 2D dentition image with the one or more target locations.

[0299] Example embodiment 4 may further extend any of example embodiments 1 -3. In example embodiment 4, determining that the first expected auxiliary component is missing in the 2D dentition image comprises determining that a target location associated with the first expected auxiliary component of the one or more target locations does not correspond to a location of the one or more locations of the auxiliary components of the 2D dentition image.

[0300] Example embodiment 5 may further extend any of example embodiments 1 -4. In example embodiments, the method further comprises using an artificial intelligence (Al) model to determine the one or more locations in the 2D dentition image of the one or more auxiliary components, wherein the Al model indicates, for each location of the one or more locations, additional component information comprising at least one of: a tooth label corresponding to the respective location; or a component type for the respective auxiliary component associated with the respective location.

[0301] Example embodiment 6 may further extend any of example embodiments 1 -5. In example embodiment 6, the method further comprises generating a first item of a training dataset for an Al model used to determine whether the one or more expected auxiliary components are present in the 2D dentition image, wherein the first item comprises: a training input including a second 2D dentition image; and a target output including one or more locations of auxiliary components depicted in the second 2D dentition image and tooth labels corresponding one or more locations, wherein the tooth labels are derived from registering a second 3D dentition model that corresponds to the second 2D dentition image to the second 3D dentition model.Attorney Docket No.: 28510.983 (L0820PCT)

[0302] Example embodiment 7 may further extend any of example embodiments 1 -6. In example embodiment 7, the method further comprises: obtaining a second item of the training dataset for the Al model used to determine whether the one or more expected auxiliary components are present in the 2D dentition image, wherein the second item comprises a target output indicating one or more locations of auxiliary components in a third 2D dentition image; obtaining a third 3D dentition model corresponding to the third 2D dentition image, wherein the third 3D dentition model indicates one or more target locations of expected auxiliary components of the third 3D dentition model; determining that the one or more locations of the target output do not match the one or more target locations indicated by the third 3D dentition model; and removing the second item from the training dataset.

[0303] Example embodiment 8 may further extend any of example embodiments 1 -7. In example embodiments, each expected auxiliary component of the one or more expected auxiliary components comprises at least one of an orthodontic attachment or an orthodontic button.

[0304] Example embodiment 9 is a system comprising: one or more processors; and a memory coupled to the one or more processors, the memory storing computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: obtaining a two-dimensional (2D) dentition image comprising a depiction of a patient's dentition, obtaining a three-dimensional (3D) dentition model comprising a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components, registering the 3D dentition model to the 2D dentition image, determining whether the one or more expected auxiliary components are present in the 2D dentition image based at least in parton information from the 3D dentition model, and outputting an indication on a user interface responsive to a first expected auxiliary component of the one or more expected auxiliary components missing from the 2D dentition image.

[0305] Example embodiment 10 may further extend example embodiment 9. In example embodiment 10, the computer-implemented method further comprises: generating a binary mask based on the registered 3D dentition model, wherein the binary mask comprises data indicating one or more locations in the 2D dentition image corresponding to the one or more expected auxiliary components; and modifying the 2D dentition image based on the binary mask.

[0306] Example embodiment 11 may further extend any of example embodiments 9-10. In example embodiment 11, an artificial intelligence (Al) model uses the 2D dentition image as input to generate output data indicating one or more locations of auxiliary components in the 2D dentition image; and the output data comprises a bounding box around each respective location of the one or more locations of the auxiliary components.

[0307] Example embodiment 12 may further extend any of example embodiments 9-11. In example embodiment 12, the 2D dentition image further comprises one or more auxiliary components;Attorney Docket No.: 28510.983 (L0820PCT)and determining whether the one or more expected auxiliary components are present in the 2D dentition image comprises matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image.

[0308] Example embodiment 13 may further extend any of example embodiments 9-12. In example embodiment 13, the 3D dentition model indicates the one or more expected auxiliary components on one or more teeth of the patient atone or more target locations; and the 2D dentition image indicates the one or more auxiliary components at one or more locations in the 2D dentition image; and matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image comprises comparing the one or more locations of the respective auxiliary components in the 2D dentition image with the one or more target locations.

[0309] Example embodiment 14 may further extend any of example embodiments 9-13. In example embodiment 14, determining that the first expected auxiliary component is missing in the 2D dentition image comprises determining that a target location associated with the first expected auxiliary component of the one or more target locations does not correspond to a location of the one or more locations of the auxiliary components of the 2D dentition image.

[0310] Example embodiment 15 is a non-transitory computer-readable storage medium with instructions stored thereon, wherein the instructions, when executed by one or more processors, perform a computer-implemented method comprising: obtaining a two-dimensional (2D) dentition image comprising a depiction of a patient's dentition; obtaining a three-dimensional (3D) dentition model comprising a depiction of the patient's dentition, wherein the 3D dentition model indicates one or more expected auxiliary components; registering the 3D dentition model to the 2D dentition image; determining whether the one or more expected auxiliary comp...

Claims

Attorney Docket No.: 28510.983 (L0820PCT)CLAIMS:What is claimed is:

1. A system, comprising:one or more processors; anda memory coupled to the one or more processors, the memory storing computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising:obtaining a two-dimensional (2D) dentition image comprising a depiction of a patients dentition,obtaining a three-dimensional (3D) dentition model comprising a depiction of the patients dentition, wherein the 3D dentition model indicates one or more expected auxiliary components,registering the 3D dentition model to the 2D dentition image,determining whether the one or more expected auxiliary components are present in the 2D dentition image based at least in part on information from the 3D dentition model, and in response to determining that a first expected auxiliary component is missing from the 2D dentition image, outputting an indication that the first expected auxiliary component is missing on a user interface.

2. The system of claim 1 , wherein the computer-implemented method further comprises:generating a binary mask based on the registered 3D dentition model, wherein the binary mask comprises data indicating one or more locations in the 2D dentition image corresponding to the one or more expected auxiliary components; andmodifying the 2D dentition image based on the binary mask.

3. The system of claim 1 or 2, wherein:the 2D dentition image further comprises one or more auxiliary components;Attorney Docket No.: 28510.983 (L0820PCT)an artificial intelligence (Al) model uses the 2D dentition image as input to generate output data indicating locations of the one or more auxiliary components in the 2D dentition image; andthe output data comprises a bounding box around each respective location of the one or more locations of the auxiliary components.

4. The system of claims 1-3, wherein:the 2D dentition image further comprises one or more auxiliary components; and determining whether the one or more expected auxiliary components are present in the 2D dentition image comprises matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image.

5. The system of claim 4, wherein:the 3D dentition model indicates the one or more expected auxiliary components on one or more teeth of the patient atone or more target locations;the 2D dentition image indicates the one or more auxiliary components atone or more locations in the 2D dentition image; andmatching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image comprises comparing the one or more locations of the respective auxiliary components in the 2D dentition image with the one or more target locations.

6. The system of claim 5, wherein determining that the first expected auxiliary component is missing in the 2D dentition image comprises determining that a target location associated with the first expected auxiliary component of the one or more target locations does not correspond to a location of the one or more locations of the auxiliary components of the 2D dentition image.

7. The system of claims 4-5, the computer-implemented method further comprising:Attorney Docket No.: 28510.983 (L0820PCT)using an artificial intelligence (Al) model to determine the one or more locations of the one or more auxiliary components in the 2D dentition image, wherein the Al model indicates, for each location of the one or more locations, additional component information comprising at least one of:a tooth label corresponding to the respective location; ora component type for the respective auxiliary component associated with the respective location.

8. The system of claims 1-7, the computer-implemented method further comprising:generating a first item of a training dataset for an Al model used to determine whetherthe one or more expected auxiliary components are present in the 2D dentition image, wherein the first item comprises:a training input including a second 2D dentition image; anda target output including one or more locations of auxiliary components depicted in the second 2D dentition image and tooth labels corresponding to the one or more locations, wherein the tooth labels are derived from registering a second 3D dentition model that corresponds to the second 2D dentition image to the second 2D dentition image.

9. The system of claim 8, the computer-implemented method further comprising:obtaining a second item of the training dataset for the Al model used to determine whetherthe one or more expected auxiliary components are present in the 2D dentition image, wherein the second item comprises a target output indicating one or more locations of auxiliary components in a third 2D dentition image;obtaining a third 3D dentition model corresponding to the third 2D dentition image, wherein the third 3D dentition model indicates one or more target locations of expected auxiliary components of the third 3D dentition model;determining that the one or more locations of the target output do not match the one or more target locations indicated by the third 3D dentition model; andremoving the second item from the training dataset.Attorney Docket No.: 28510.983 (L0820PCT)10. The system of claims 1-9, wherein each expected auxiliary component of the one or more expected auxiliary components comprises at least one of an orthodontic attachment or an orthodontic button.

11. The system of claims 1-10, wherein the system comprises a mobile computing device with a camera that captures the 2D dentition image.

12. A method, comprising:obtaining a two-dimensional (2D) dentition image comprising a depiction of a patients dentition; transmitting the 2D dentition image to a server device;receiving, from the server device, an indication of a first expected auxiliary component that is missing from the 2D dentition image,wherein the information has been generated by the server device by:obtaining a three-dimensional (3D) dentition model comprising a depiction of the patients dentition, wherein the 3D dentition model indicates one or more expected auxiliary components;registering the 3D dentition model to the 2D dentition image; anddetermining whether one or more expected auxiliary components are present in the 2D dentition image based at least in parton information from the 3D dentition model, wherein a determination is made that the first expected auxiliary component is missing from the 2D dentition image; andoutputting the indication on a user interface.

13. The method of claim 12, wherein:the 2D dentition image further comprises one or more auxiliary components; and determining whether the one or more expected auxiliary components are present in the 2D dentition image comprises matching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image.Attorney Docket No.: 28510.983 (L0820PCT)14. The method of claim 13, wherein:the 3D dentition model indicates the one or more expected auxiliary components on one or more teeth of the patient atone or more target locations;the 2D dentition image indicates the one or more auxiliary components atone or more locations in the 2D dentition image; andmatching one or more of the expected auxiliary components of the registered 3D dentition model to the one or more auxiliary components of the 2D dentition image comprises comparing the one or more locations of the respective auxiliary components in the 2D dentition image with the one or more target locations.

15. The method of claim 14, wherein determining that the first expected auxiliary component is missing in the 2D dentition image comprises determining that a target location associated with the first expected auxiliary component of the one or more target locations does not correspond to a location of the one or more locations of the auxiliary components of the 2D dentition image.

16. The method of claims 14-15, wherein the server device performs further operations comprising:using an artificial intelligence (Al) model to determine the one or more locations of the one or more auxiliary components in the 2D dentition image, wherein the Al model indicates, for each location of the one or more locations, additional component information comprising at least one of:a tooth label corresponding to the respective location; ora component type for the respective auxiliary component associated with the respective location.

17. The method of claims 12-16, wherein the server device performs further operations comprising:generating a first item of a training dataset for an Al model used to determine whether the one or more expected auxiliary components are present in the 2D dentition image, wherein the first item comprises:a training input including a second 2D dentition image; andAttorney Docket No.: 28510.983 (L0820PCT)a target output including one or more locations of auxiliary components depicted in the second 2D dentition image and tooth labels corresponding to the one or more locations, wherein the tooth labels are derived from registering a second 3D dentition model that corresponds to the second 2D dentition image to the second 2D dentition image.

18. The method of claim 17, wherein the server device performs further operations comprising:obtaining a second item of the training dataset for the Al model used to determine whether the one or more expected auxiliary components are present in the 2D dentition image, wherein the second item comprises a target output indicating one or more locations of auxiliary components in a third 2D dentition image;obtaining a third 3D dentition model corresponding to the third 2D dentition image, wherein the third 3D dentition model indicates one or more target locations of expected auxiliary components of the third 3D dentition model;determining that the one or more locations of the target output do not match the one or more target locations indicated by the third 3D dentition model; andremoving the second item from the training dataset.

19. The method of claims 12-18, wherein each expected auxiliary component of the one or more expected auxiliary components comprises at least one of an orthodontic attachment or an orthodontic button.

20. The method of claims 12-19, wherein the obtaining, transmitting, receiving, and outputting are performed by a mobile computing device comprising a camera.

Images