Digital impression method and system based on head-mounted display, and head-mounted display

Abstract

The present application discloses a digital impression method and system based on a head-mounted display, and a head-mounted display. The method comprises: splicing and matching a plurality of feature point data frames collected in real time by a head‑mounted display camera to obtain frame point data of coded scan bodies; on the basis of a feature point in the frame point data of the coded scan bodies that matches standard data, splicing the corresponding standard data in the head‑mounted display, so as to generate first spliced data; and when the number of scan bodies in the first spliced data is inconsistent with the number of mounted scan bodies, generating a corresponding adjustment suggestion on the basis of an unmatched feature point in the frame point data of the coded scan bodies. In the method, the head‑mounted display is used to replace an extraoral scanner, and the first spliced data is generated using the feature point matching the standard data in the frame point data of the coded scan bodies, so as to assist in tracking the scanning progress of the coded scan bodies. In addition, the adjustment suggestion is generated using the unmatched feature point, so as to assist a dentist in detecting whether any coded scan body has not been scanned, thereby lowering an operational threshold of the impression process.

Description

Digital modeling methods and systems based on head-mounted displays, and head-mounted displays themselves. Cross-references to related applications

[0001] This application claims priority to Chinese Patent Application No. 202510397849.5, filed on March 28, 2025, entitled "Digital Imaging Method, System and Head-Mounted Display Based on Head-Mounted Display", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of digital modeling technology, and in particular to a digital modeling method, system, and head-mounted display based on a head-mounted display. Background Technology

[0003] Dental implants are a dental restoration technique that is an effective restorative method for tooth loss, functional impairment, aesthetic needs, or other specific circumstances. The entire process of dental implant surgery is as follows: local anesthesia, incision of the gums, exposure of the alveolar bone, preparation of the implant socket in the alveolar bone (i.e., drilling a hole that matches the implant), implant placement, waiting for osseointegration (after implantation, a period of time is required (usually 3-6 months) for the implant to gradually integrate with and stabilize with the surrounding bone tissue), installation of the healing abutment, impression taking, crown fabrication, and crown installation. The purpose of the impression taking step is to determine the positional relationship of the implants in the mouth, as well as the relative positional relationship between multiple implants. Based on these positional relationships, the bridge and crown can be fabricated.

[0004] Depending on the molding method, molding can be divided into traditional molding and digital molding. Traditional molding typically uses alginate to create a primary mold, which is then used to create individual trays for secondary molding. Traditional molding technology has some limitations, such as cumbersome procedures, time-consuming processes, difficulty in controlling molding pressure, high technical sensitivity, and the possibility of mold deformation and difficulty in preservation. Compared to digital molding technology, traditional molding technology may be less precise and efficient.

[0005] Digital impression technology uses a digital dental impression instrument to take impressions for implant placement. Since the implant is located inside the alveolar bone and cannot be directly scanned, a scanning rod needs to be installed on the implant. The impression instrument scans the scanning rod to obtain its position, which in turn determines the position of the implant.

[0006] Scanning poles are further divided into ordinary scanning poles and coded scanning poles. These two types of scanning poles have different scanning modes and accuracies. Ordinary scanning poles are small rods made of metal or other materials, generally cylindrical, and have features such as steps and beveled surfaces to facilitate feature recognition during impression scanning. During scanning, the scanning pole is installed on the intraoral implant and scans along with the teeth to obtain grid data. Due to the small size of ordinary scanning poles and their generally metal material, they are highly reflective, often resulting in poor data quality. Furthermore, when multiple ordinary scanning poles are scanned together with the teeth, the overall deformation of the teeth data may be significant, leading to inaccurate positional relationships between the multiple ordinary scanning poles, with errors reaching hundreds of micrometers. In actual use, the digital impression taking process with ordinary scanning poles is as follows: 1. Scan the gingiva using an intraoral scanner to obtain a three-dimensional model of the gingiva; 2. The patient wears a mouth diffuser, and the ordinary scanning pole is connected to the implant or abutment, followed by scanning.

[0007] The coded scanning rod has feature points (as shown in Figure 8). During scanning, only the feature points are identified; it is not necessary to scan the overall model data of the rod. Multiple rods are brought together by a crossbar, eliminating the need for splicing transitions using gingival data. Therefore, the final positioning accuracy of the rod can reach tens of micrometers. In actual use, the digital impression process of the coded scanning rod is as follows: 1. Use an intraoral scanner to scan the gingiva to obtain a three-dimensional model A of the gingiva; 2. The patient wears a mouth diffuser, and the coded scanning rod is connected to the implant or abutment. Then, an intraoral scanner (or an extraoral scanner) is used to scan the feature point data B on the rod to determine the relative positional relationship between the rods; 3. Using an intraoral scanner (or an extraoral scanner), a small portion of the gingival data C and feature point data D are scanned simultaneously to align the three-dimensional model A with the feature point data B, thus determining the positional relationship between the rod and the gingiva. Step 3 and step 2 can be performed simultaneously, i.e., scanning the gingival data C and feature point data B at the same time. The gingival data C is used to splice the three-dimensional model A, while the feature point data B is directly used as the final rod data.

[0008] Traditional impression taking is complex, and its accuracy is greatly affected by the operator's experience level. The process is difficult to digitize. The digital impression taking process of ordinary scanning rod has solved the shortcomings of traditional impression taking, but the accuracy is still not good enough. Therefore, dentists need to have a high level of professional knowledge to effectively complete the impression taking work. Summary of the Invention

[0009] This application provides a digital impression taking method, system, and head-mounted display based on a head-mounted display, with the aim of using the head-mounted display to assist dentists in taking impressions, thereby reducing the operational threshold for impression taking.

[0010] To achieve the above objectives, this application provides the following technical solution:

[0011] A digital modeling method based on a head-mounted display includes: stitching and matching multi-frame feature point data acquired in real time by a head-mounted display camera to obtain coded scanning rod frame point data; stitching corresponding standard data in the head-mounted display based on feature points in the coded scanning rod frame point data that match standard data to generate first stitched data; and generating corresponding adjustment suggestions based on unmatched feature points in the coded scanning rod frame point data when the number of scanning rods in the first stitched data does not match the number of scanning rods installed.

[0012] Optionally, the method further includes: obtaining data to be stitched based on oral cavity scanning data acquired by an intraoral scanner; obtaining oral cavity data of multiple regions acquired by a head-mounted display camera; the oral cavity data includes feature point data of coded scanning rods within the regions, and the oral cavity data is image data and / or model data; stitching the data to be stitched with multiple oral cavity data respectively, so that the data to be stitched is aligned with the multiple oral cavity data; stitching the feature point data in the multiple oral cavity data with the frame point data of the coded scanning rod respectively, so that the data to be stitched is aligned with the frame point data of the coded scanning rod.

[0013] Optionally, the data to be stitched can be stitched together with multiple oral cavity data to align the data to be stitched with the multiple oral cavity data. This includes: for each oral cavity data, determining whether the oral cavity data matches the data to be stitched; if the oral cavity data matches the data to be stitched, stitching the data to be stitched together with the oral cavity data to align the data to be stitched with the oral cavity data.

[0014] Optionally, the method also includes: if the oral cavity data does not match the data to be stitched, triggering the head-mounted camera to acquire new oral cavity data from multiple regions; the new oral cavity data is used to re-match with the data to be stitched.

[0015] Optionally, based on feature points in the coded scanning rod frame point data that match standard data, the corresponding standard data is stitched together in the head-mounted display to generate the first stitched data. This includes: when obtaining feature point data for each frame, determining the target coded scanning rod that matches the feature points in the historical frame feature point data; for each feature point in the feature point data of each frame, if the feature point belongs to the target coded scanning rod, marking the feature point as a matched point; if the feature point does not belong to the target coded scanning rod, marking the feature point as an unmatched point; matching the unmatched point with the feature points of the coded scanning rods pre-stored in the scanning rod database to obtain the corresponding matching result; if the matching result is successful, remarking the unmatched point as a matched point; based on the matched points in the feature point data of each frame and the feature points in the historical frame feature point data, determining the feature points in the coded scanning rod frame point data that match the standard data; and stitching the corresponding standard data in the head-mounted display based on the feature points in the coded scanning rod frame point data that match the standard data to generate the first stitched data.

[0016] Optionally, when the number of scanning poles in the first stitched data does not match the required number of scanning poles to be installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning pole frame point data. These suggestions include: determining the attribute information of the unmatched feature points in the coded scanning pole frame point data; the attribute information includes at least position, distance, clarity, and frequency of occurrence; and generating corresponding adjustment suggestions based on the attribute information of the unmatched feature points. These adjustment suggestions include adjusting the patient's mouth posture, adjusting the spacing between the coded scanning poles, prompting a re-download of the scanning pole database, and / or prompting an adjustment of the distance between the head-mounted display and the coded scanning poles.

[0017] Optionally, the method also includes: displaying matched points and unmatched points in the head-mounted display; wherein the display effect of matched points is different from that of unmatched points.

[0018] Optionally, the standard data includes standard scan bar data, which is used to identify the relative positional relationships between multiple coded scan bars.

[0019] Optionally, the adjustment suggestion is used to prompt the head-mounted display wearer to perform the corresponding adjustment action so that the coded scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coded scanning rod.

[0020] Optionally, the data to be spliced ​​includes gingival model data, which is used to characterize the corresponding gingival model.

[0021] Optionally, the oral data may also include pre-added spot features on the region, which may include optical features projected onto the region using third-party optical equipment.

[0022] Optionally, the first splicing data is used to determine the relative positional relationship between the individual coded scanning rods.

[0023] A head-mounted display (HUD) includes: a data determination unit configured to stitch together and match multi-frame feature point data acquired in real time by the HUD camera to obtain coded scanning rod frame point data; a data stitching unit configured to stitch corresponding standard data in the HUD based on feature points in the coded scanning rod frame point data that match standard data, generating first stitched data; and a suggestion generation unit configured to generate corresponding adjustment suggestions based on unmatched feature points in the coded scanning rod frame point data when the number of scanning rods in the first stitched data does not match the number of scanning rods installed.

[0024] A head-mounted display-based digital impression system includes an intraoral scanner and a head-mounted display. The intraoral scanner is configured to acquire oral cavity scan data of a patient, and the head-mounted display is configured to perform the aforementioned head-mounted display-based digital impression method.

[0025] A storage medium comprising a stored program, wherein the program is executed by a processor during runtime as a head-mounted display-based digital image acquisition method.

[0026] The technical solution provided in this application stitches and matches multi-frame feature point data acquired in real time by a head-mounted display camera to obtain coded scanning rod frame point data. Based on the feature points in the coded scanning rod frame point data that match standard data, the corresponding standard data is stitched together in the head-mounted display to generate first stitched data. When the number of scanning rods in the first stitched data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. This application uses a head-mounted display to replace an extraoral scanner to obtain coded scanning rod frame point data, uses the feature points in the coded scanning rod frame point data that match standard data to generate first stitched data to assist in tracking the scanning progress of the coded scanning rods, and uses the unmatched feature points to generate adjustment suggestions to assist dentists in detecting whether any coded scanning rods have been missed, thereby reducing the operational threshold of the impression taking process. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 is a schematic diagram of the architecture of a digital modeling system based on a head-mounted display provided in an embodiment of this application;

[0029] Figure 2 is a flowchart illustrating a digital modeling method based on a head-mounted display provided in an embodiment of this application;

[0030] Figure 3 is a flowchart illustrating another digital modeling method based on a head-mounted display provided in an embodiment of this application;

[0031] Figure 4 is a flowchart illustrating another digital modeling method based on a head-mounted display provided in an embodiment of this application;

[0032] Figure 5 is a flowchart illustrating another digital modeling method based on a head-mounted display provided in an embodiment of this application;

[0033] Figure 6 is a flowchart illustrating another digital modeling method based on a head-mounted display provided in an embodiment of this application;

[0034] Figure 7 is a schematic diagram of the architecture of a head-mounted display provided in an embodiment of this application;

[0035] Figure 8 is a schematic diagram of an encoding scanning rod provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] Figure 1 shows a schematic diagram of the architecture of a digital modeling system based on a head-mounted display provided in an embodiment of this application, which includes the following modules.

[0039] The intraoral scanner 100 and head-mounted display 200 are configured to acquire gingival scan data of a patient, and the head-mounted display 200 is configured to perform the head-mounted display-based digital impression method provided in this application.

[0040] In some examples, the head-mounted display 200 may be of types including, but not limited to, head-mounted displays (such as virtual reality devices, augmented reality devices, and hybrid display devices) and smart glasses. Generally, the head-mounted display 200 and its operating system provide the basic hardware and computing resources required for 3D modeling, including but not limited to cameras (such as binocular cameras), LiDAR, and structured light scanning components.

[0041] It should be noted that in this embodiment of the application, a head-mounted display 200 is used to replace an extraoral scanner to perform scanning work on the coded scanning rod on the patient's gums. For this purpose, it is necessary to ensure that the camera of the head-mounted display 200 has reliable accuracy. If the head-mounted display 200 has not been calibrated or a long time has passed since the last camera calibration, the head-mounted display 200 needs to be calibrated before using it to assist the dentist in taking impressions.

[0042] In some examples, the purpose of camera calibration is to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. When using a binocular reconstruction algorithm to reconstruct and stitch together feature point data from multiple frames to obtain coded scanning rod frame point data, the intrinsic and extrinsic parameters of the head-mounted display camera are required.

[0043] In a possible implementation, the steps for camera calibration are as follows.

[0044] Step 1: Prepare the calibration object, usually using a calibration plate with known size and pattern (e.g., checkerboard or dot array).

[0045] Step 2: Trigger the head-mounted display camera to capture calibration images. Specifically, place the calibration board at different angles and positions, and capture multiple calibration images to ensure that the calibration board is clearly visible in each calibration image.

[0046] Step 3: Process the calibration image. Specifically, use image processing software (such as OpenCV) to detect the calibration image to obtain feature points.

[0047] Step 4: Calculate camera parameters (i.e., intrinsic and extrinsic parameters) using feature points. Specifically, based on the obtained feature points, use a calibration algorithm to calculate the intrinsic and extrinsic parameters of the head-mounted display camera. The intrinsic and extrinsic parameters include, but are not limited to, focal length, principal point coordinates, distortion coefficients, etc.

[0048] Step 5: Verify the calibration results. Specifically, verify the accuracy of the calibration results (internal and external parameters) by using indicators such as reprojection error. If the error is large, retake the calibration image or adjust the calibration parameters.

[0049] After the head-mounted display camera is calibrated, the dentist can wear the head-mounted display to take impressions. The head-mounted display camera will capture the image from the dentist's current perspective and display the image on the screen. In this embodiment, the head-mounted display-based digital impression taking method guides the dentist to use the head-mounted display to replace the external scanner, realizing the operation of the scanning coding scanning rod. Please refer to the steps shown in Figure 2 and the corresponding explanation.

[0050] Figure 2 shows a flowchart of a digital modeling method based on a head-mounted display provided in an embodiment of this application, including the following steps.

[0051] S201: The multi-frame feature point data collected in real time by the head-mounted display camera is stitched together and matched to obtain the coded scanning rod frame point data.

[0052] In this process, after the patient wears a mouth flare and the coded scanning rod is installed on the implant or abutment, the head-mounted display can reconstruct and stitch together multiple frames of feature point data through feature point reconstruction and matching algorithms to obtain the frame point data of the coded scanning rod. The principle of this implementation can be understood as using multiple images or multiple sets of local data (i.e., frame feature point data, or scanning rod image frames) to form a large image or complete data (i.e., coded scanning rod frame point data, complete scanning rod image). Each time, the head-mounted display will only show a part of the large image or complete data, so that the dentist can find the location of all images or data in as few attempts as possible.

[0053] In some examples, the encoded scan lever frame point data includes information such as the position of feature points on the encoded scan lever and whether they match. Generally speaking, the encoded scan lever frame point data is empty before the first frame of feature point data is obtained.

[0054] In a possible implementation, the head-mounted display acquires images of the scanning rod through a binocular camera. Since the coded scanning rod carries some specially designed marker points (which can be regarded as feature points), the marker points of the coded scanning rod have encoded information. During the intraoral scanning process, the corresponding position coordinates can be obtained. Therefore, binocular stereo vision and 3D reconstruction algorithms can be used to reconstruct and extract the feature points on the coded scanning rod to obtain real-time multi-frame feature point data, thereby achieving precise positioning of the scanning rod.

[0055] It should be noted that when obtaining feature point data of any frame, if the current encoded scan lever frame point data is not empty, the current frame feature point data and the current encoded scan lever frame point data (obtained by splicing and matching based on the feature point data of historical frames) need to be spliced ​​and matched to obtain more complete encoded scan lever frame point data.

[0056] S202: Based on the feature points in the coded scanning rod frame point data that match the standard data, the corresponding standard data is stitched together in the head-mounted display to generate the first stitched data.

[0057] The standard data includes standard scan bar data, which is used to identify the relative positional relationship between multiple coded scan bars.

[0058] It is important to note that coded scan points are unique. Only a specific number of feature points, such as six or more, need to be identified to match the corresponding coded scan point in the scan point database, thus determining its position and orientation. Therefore, the coded scan point frame data inevitably contains redundant feature points. For the real-time updated coded scan point frame data, feature points in the current frame feature point data may be redundant feature points of previously matched coded scan points. Therefore, we can pre-traverse the target coded scan points that have already been matched. If a feature point in the current frame feature point data belongs to a target coded scan point, then that feature point can be marked as a matched point, thereby determining the feature points in the coded scan point frame data that match the standard data.

[0059] Optionally, the process of stitching the corresponding standard data in the head-mounted display based on the feature points in the coded scanning rod frame point data that match the standard data to generate the first stitched data can be found in the steps shown in Figure 3 and the corresponding explanations.

[0060] S203: When the number of scanning rods in the first stitched data does not match the number of scanning rods installed, generate corresponding adjustment suggestions based on the unmatched feature points in the coded scanning rod frame point data.

[0061] The adjustment suggestions are used to prompt the wearer of the headset to perform corresponding adjustment actions so that the frame point data can add feature points that match the missing data. The missing data includes the standard data part corresponding to the missing coded scanning rod.

[0062] It is understandable that when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, it can be assumed that some of the coded scanning rods installed in the patient's gums have failed to be successfully modeled, that is, there are scanning rods that have not been spliced. The coded scanning rods that have not been successfully modeled can be regarded as missing coded scanning rods.

[0063] In some examples, the reasons for missing coded scanning rods during the scanning process include, but are not limited to: feature points on the missing coded scanning rod being occluded, or the head-mounted display camera failing to capture feature points on the missing coded scanning rod due to shooting angle issues.

[0064] Therefore, when the number of scanning rods in the first splicing data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. This can address the reasons for missing coded scanning rods during the scanning process and propose corresponding solutions to help dentists complete the modeling of missing coded scanning rods in a timely and effective manner.

[0065] Optionally, when the number of scanning rods in the first stitched data does not match the number of scanning rods installed, the process of generating corresponding adjustment suggestions based on the unmatched feature points in the coded scanning rod frame point data can be found in the steps shown in Figure 4 and the corresponding explanations.

[0066] It is important to note that during the digital impression process, in addition to scanning the frame point data of all coded scanning rods on the patient's gums, it is also necessary to align the patient's gum model with the frame point data of the coded scanning rods in order to determine the positional relationship between the coded scanning rods and the patient's gums.

[0067] Optionally, the process of aligning the patient's gingival model with the frame point data can be achieved using a head-mounted display, as shown in Figure 5 and the corresponding explanation.

[0068] The process described in S201-S203 above uses a head-mounted display to replace the external scanner to obtain coded scanning rod frame point data. The first stitched data is generated using feature points in the coded scanning rod frame point data that match the standard data to help track the scanning progress of the coded scanning rod. Furthermore, the corresponding adjustment suggestions are generated using feature points in the coded scanning rod frame point data that do not match to help dentists detect whether any coded scanning rods have been missed, thereby effectively reducing the operational threshold of the impression taking process.

[0069] Figure 3 shows a flowchart of another digital modeling method based on a head-mounted display provided in this application embodiment, which includes the following steps.

[0070] S301: When obtaining feature point data for each frame, determine the target encoded scan rod that matches the feature points in the historical frame feature point data.

[0071] Specifically, for each frame of feature point data, the target coded scan bar that matches the feature points in the historical frame feature point data can be understood as the target coded scan bar that matches the current coded scan bar frame point data. In other words, at least one feature point in the current coded scan bar frame point data belongs to a target coded scan bar, and the number of target coded scan bars is one or more.

[0072] In a possible implementation, the target coding scan bar can be regarded as a coding scan bar that has been determined to match within a historical period.

[0073] S302: For each feature point in each frame of feature point data, determine whether the feature point belongs to the target encoding scan rod.

[0074] If the feature point belongs to the target encoding scan bar, execute S303; if the feature point does not belong to the target encoding scan bar, execute S304.

[0075] S303: Mark the feature point as a matched point.

[0076] After executing S303, S307 is executed.

[0077] S304: Mark the feature point as an unmatched point.

[0078] After executing S304, S305 is executed.

[0079] Optionally, matched and unmatched points can also be displayed in the head-mounted display, with matched points displayed differently from unmatched points.

[0080] In some examples, matched points may be displayed as green dots, while unmatched points may be displayed as red dots.

[0081] It can be understood that displaying matched and unmatched points in the head-mounted display allows dentists to know the scanning effect of feature points on the coded scanning rod in real time.

[0082] S305: Match the unmatched points with the feature points of the coded scan rods pre-stored in the scan rod database to obtain the corresponding matching results.

[0083] The process involves matching unmatched points with feature points of pre-stored coded scan bars in the scan bar database. If a new coded scan bar in the database matches the unmatched point, the match is considered successful. If no new coded scan bar matches the unmatched point, the match is considered unsuccessful. For the determined set of unmatched points, each unmatched point in the set is sequentially matched with the feature points in the scan bar database until no new coded scan bar can be determined from the database.

[0084] S306: If the matching result is successful, remark the unmatched point as a matched point.

[0085] After executing S306, S307 is executed.

[0086] S307: Based on the matched points in the feature point data of each frame and the feature points in the feature point data of historical frames, determine the feature points in the coded scan rod frame point data that match the standard data.

[0087] Among them, based on the matched points in the feature point data of each frame, and the feature points in the feature point data of historical frames that match the target coding scan rod, the feature points in the coding scan rod frame point data that match the standard data are determined, which can effectively determine the matched points in the real-time updated coding scan rod frame point data.

[0088] S308: Based on the feature points in the coded scanning rod frame point data that match the standard data, the corresponding standard data is stitched together in the head-mounted display to generate the first stitched data.

[0089] In this process, feature points in the frame point data of the coded scanning rod that match the standard data are stitched together in the head-mounted display to generate the first stitched data. The purpose is to stitch the corresponding standard data into the scanning rod image displayed on the head-mounted display, so as to identify the relative positional relationship between multiple coded scanning rods in the scanning rod image.

[0090] It should be noted that stitching the corresponding standard data onto the scanning bar image displayed on the head-mounted display helps dentists track the scanning progress of the coded scanning bar. Generally speaking, once the standard data corresponding to each coded scanning bar installed on the patient's gums has been determined and stitched together, the first stitched data can be saved to determine the relative positional relationship between the various coded scanning bars.

[0091] The process described in S301-S308 above utilizes the matched points in the real-time updated coded scanning rod frame point data to stitch the corresponding standard data in the head-mounted display, generating the first stitched data, thereby assisting the dentist in tracking the scanning progress of the coded scanning rod.

[0092] Figure 4 shows a flowchart of another digital modeling method based on a head-mounted display provided in this application embodiment, which includes the following steps.

[0093] S401: When the number of scanning rods in the first splicing data does not match the number of scanning rods installed, determine the attribute information of the unmatched feature points in the encoded scanning rod frame point data.

[0094] The attribute information includes at least location, distance, sharpness, and frequency of occurrence. During the stitching process of the first data assembly, there are unmatched feature points (i.e., unmatched points) in the coded scan rod frame point data. These unmatched points did not participate in the stitching process of the first data assembly. Therefore, by analyzing the attribute information of the unmatched points, the reason for the missing coded scan rods during the scanning process can be discovered.

[0095] S402: Generate corresponding adjustment suggestions based on the attribute information of unmatched feature points.

[0096] The adjustment suggestions include adjusting the patient's mouth posture, adjusting the spacing between the coded scanning rods, prompting the user to re-download the scanning rod database, and / or prompting the user to adjust the distance between the head-mounted display and the coded scanning rods.

[0097] It should be noted that, based on the attribute information of the unmatched feature points, the reasons for the mismatch between the unmatched points and the standard data can be analyzed and determined. These reasons may include incorrect patient mouth posture, excessive distance between two adjacent coded scanning rods, excessive distance between the head-mounted display and the coded scanning rod on the patient's gums, and the unmatched points not conforming to the feature point standards in the scanning rod database.

[0098] In some examples, based on the attribute information of unmatched feature points, it is determined that the patient's mouth posture is incorrect, so adjustment suggestions can be generated to prompt the patient to adjust their mouth posture.

[0099] In some examples, based on the attribute information of unmatched feature points, it is determined that the spacing between two adjacent coded scanning rods is too far. Therefore, adjustment suggestions can be generated to prompt dentists to adjust the spacing between the coded scanning rods so that the two adjacent coded scanning rods are closer to each other.

[0100] In some examples, based on the attribute information of unmatched feature points, it is determined that the head-mounted display is too far from the coded scanning bar on the patient's gums. Therefore, adjustment suggestions can be generated to prompt the dentist to move closer to the coded scanning bar to shorten the distance between the head-mounted display and the coded scanning bar.

[0101] In some examples, based on the attribute information of unmatched feature points, it is determined that the unmatched points do not meet the feature point standards in the scan bar database. For example, if the coded scan bars of another set of coded scan bars are scanned together, the missing coded scan bars cannot be identified. Therefore, adjustment suggestions can be generated to prompt the user to download the scan bar database of another set of coded scan bars again.

[0102] The process described in S401-S402 above guides dentists to perform corresponding adjustment actions based on the attribute information of unmatched feature points in the coded scanning rod frame point data. This ensures that all coded scanning rods on the patient's gums can complete the corresponding standard data splicing, achieving complete modeling of each coded scanning rod.

[0103] Figure 5 shows a flowchart of another digital modeling method based on a head-mounted display provided in this application embodiment, which includes the following steps.

[0104] S501: Obtain the data to be stitched based on the oral cavity scan data collected by the intraoral scanner.

[0105] The data to be spliced ​​includes gingival model data, which is used to characterize the corresponding gingival model.

[0106] It is important to note that if the patient has edentulous gums, due to the lack of distinctive features in edentulous gums, an intraoral scanner can be used to scan the patient's oral cavity using projected structured light to obtain oral scan data. A transmission link is pre-established between the intraoral scanner and the head-mounted display (HMD) so that the intraoral scanner can transmit oral scan data to the HMD via this link. After the HMD obtains the oral scan data, its rendering engine is used to render the oral scan data in real time to obtain the data to be stitched together.

[0107] In some examples, the headset can display real-time data of the patient's gum model to help dentists understand the patient's oral cavity.

[0108] S502: Acquire oral cavity data from multiple regions captured by the head-mounted camera.

[0109] The oral cavity data includes feature point data of the coded scanning rods within the region, and the oral cavity data is image data and / or model data.

[0110] In some examples, the term "region" can be understood as a portion of the gingival area.

[0111] In the process of digital impression taking, in order to determine the positional relationship between the coded scanning rod and the patient's gums, it is also necessary to obtain a small segment of gum model (i.e., oral data) and coded scanning rod frame point data for splicing.

[0112] Optionally, the oral cavity data also includes pre-added spot features on the region, which include optical features projected onto the region using a third-party optical device. The third-party optical device has a pre-set light projection structure that can project corresponding optical features onto the region, helping the head-mounted display camera to collect oral cavity data from multiple regions.

[0113] It should be noted that because the gingiva of edentulous jaws lacks features, light spot features are added to the region using a preset light projection structure so that the oral cavity data of multiple regions collected by the head-mounted display camera has corresponding feature data. Parallel light rays and the rendering engine on the head-mounted display are used to render the feature data in the oral cavity data in real time to obtain the oral cavity data of the corresponding region.

[0114] In some examples, the third-party optical device can be a light pen. The light point projected by the light pen can be moved to the corresponding area, and then the head-mounted display can be used to collect oral data from multiple areas. At this time, the oral data includes feature point data of the gums and the coded scanning rod, so the stitching of oral data and coded scanning rod frame point data can be performed simultaneously.

[0115] Furthermore, since the oral data comes from an intraoral scanner, there is a discrepancy between the data to be stitched and the oral data (i.e., the coordinate systems of the data to be stitched and the oral data are different). In order to reduce errors, oral data from multiple regions (e.g., the right side of the gingiva, the middle of the gingiva, and the right side of the gingiva) are collected. By subdividing the oral data from multiple regions, more reference feature points are provided to ensure that the stitching between the data to be stitched and the oral data is more stable and reliable.

[0116] In some examples, the division of multiple regions can be set by technicians according to the actual situation. Specifically, multiple regions may include the left side of the gum, the middle of the gum, and the right side of the gum. By splicing different regions multiple times, more constraints are obtained, which can be used to optimize the alignment relationship between the data to be spliced ​​and the oral cavity data, so as to minimize the overall splicing error.

[0117] S503: The data to be stitched is stitched together with multiple oral cavity data respectively, so that the data to be stitched is aligned with the multiple oral cavity data.

[0118] In this process, the data to be stitched is stitched together with multiple oral cavity data to align the data to be stitched with the multiple oral cavity data. Essentially, this involves converting the multiple oral cavity data and the data to be stitched into a unified coordinate system.

[0119] Optionally, the data to be stitched can be stitched together with multiple oral cavity data to align the data to be stitched with the multiple oral cavity data. See the steps shown in Figure 6 and the corresponding explanations.

[0120] S504: The feature point data from multiple oral cavity datasets are spliced ​​and matched with the coded scanning rod frame point data to ensure that the data to be spliced ​​is aligned with the coded scanning rod frame point data.

[0121] Specifically, by stitching and matching the feature point data shown in multiple oral cavity data sets with the coded scanning rod frame point data, the feature point data in each oral cavity data set can be aligned with the coded scanning rod frame point data. Since the feature point data in each oral cavity data set is reconstructed in the head-mounted display camera coordinate system, it can be ensured that each oral cavity data set is aligned with the coded scanning rod frame point data. Furthermore, since the data to be stitched is aligned with multiple oral cavity data sets, it can be determined that the data to be stitched is aligned with the coded scanning rod frame point data to the same coordinate system.

[0122] The process described in S501-S504 above utilizes oral data from multiple regions, matching the data to be stitched with the data to be stitched, and matching the feature point data from multiple oral data with the coded scanning rod frame point data. This ensures that the data to be stitched and the coded scanning rod frame point data are aligned to the same coordinate system. Through multiple stitches on the left side of the gingiva, the middle of the gingiva, and the right side of the gingiva, more constraints are obtained, which can optimize the alignment relationship between the gingiva and the scanning rod, minimizing the overall error, thereby obtaining the positional relationship between the patient's gingiva and the coded scanning rod.

[0123] Figure 6 shows a flowchart of another digital modeling method based on a head-mounted display provided in an embodiment of this application, which includes the following steps.

[0124] S601: For each oral cavity data, determine whether the oral cavity data matches the data to be spliced.

[0125] If the oral cavity data matches the data to be assembled, then S602 is executed; if the oral cavity data does not match the data to be assembled, then S603 is executed.

[0126] S602: Combine the data to be combined with the oral cavity data to align the data to be combined with the oral cavity data.

[0127] Once the data to be stitched and the oral cavity data are successfully stitched together, they will be converted to a unified coordinate system.

[0128] S603: Triggers the head-mounted display camera to acquire new oral cavity data from multiple areas.

[0129] The new oral data is used to re-match the data to be spliced.

[0130] The processes described in S601-S603 above ensure that the data to be spliced ​​is successfully spliced ​​by re-collecting new oral data from multiple regions if any oral data does not match the data to be spliced.

[0131] Figure 7 shows a schematic diagram of the architecture of a head-mounted display provided in an embodiment of this application, including the following units.

[0132] The data determination unit 100 is configured to stitch together and match multi-frame feature point data acquired in real time by the head-mounted display camera to obtain coded scanning rod frame point data.

[0133] The data stitching unit 200 is configured to stitch the corresponding standard data in the head-mounted display based on the feature points in the coded scanning rod frame point data that match the standard data, thereby generating the first stitched data.

[0134] Optionally, the data stitching unit 200 is specifically configured to: when obtaining feature point data for each frame, determine the target coded scanning rod that matches the feature points in the historical frame feature point data; for each feature point in the feature point data of each frame, if the feature point belongs to the target coded scanning rod, mark the feature point as a matched point; if the feature point does not belong to the target coded scanning rod, mark the feature point as an unmatched point; match the unmatched point with the feature points of the coded scanning rods pre-stored in the scanning rod database to obtain the corresponding matching result; if the matching result is successful, remark the unmatched point as a matched point; based on the matched points in the feature point data of each frame and the feature points in the historical frame feature point data, determine the feature points in the coded scanning rod frame point data that match the standard data; based on the feature points in the coded scanning rod frame point data that match the standard data, stitch the corresponding standard data in the head-mounted display to generate the first stitched data.

[0135] Optionally, the standard data includes standard scan bar data, which is used to identify the relative positional relationships between multiple coded scan bars.

[0136] It is suggested that the generation unit 300 be configured to generate corresponding adjustment suggestions based on the unmatched feature points in the coded scan rod frame point data when the number of scan rods in the first stitched data does not match the number of scan rods installed.

[0137] Optionally, the generation unit 300 is specifically configured to: when the number of scanning rods in the first stitched data does not match the number of scanning rods installed, determine the attribute information of unmatched feature points in the coded scanning rod frame point data; the attribute information includes at least position, distance, clarity, and frequency of occurrence; based on the attribute information of the unmatched feature points, generate corresponding adjustment suggestions; wherein, the adjustment suggestions include adjusting the patient's mouth posture, adjusting the spacing between the coded scanning rods, prompting to re-download the scanning rod database, and / or prompting to adjust the distance between the head-mounted display and the coded scanning rods.

[0138] Optionally, the adjustment suggestion is used to prompt the head-mounted display wearer to perform the corresponding adjustment action so that the coded scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coded scanning rod.

[0139] The gingival alignment unit 400 is configured to: obtain data to be stitched based on oral scan data acquired by an intraoral scanner; obtain oral data of multiple regions acquired by a head-mounted display camera; the oral data includes feature point data of coded scanning rods within the regions, and the oral data is image data and / or model data; stitch the data to be stitched with the multiple oral data respectively to align the data to be stitched with the multiple oral data; and stitch the feature point data in the multiple oral data with the frame point data of the coded scanning rod respectively to align the data to be stitched with the frame point data of the coded scanning rod.

[0140] Optionally, the gingival alignment unit 400 is specifically configured to: for each oral data, determine whether the oral data matches the data to be spliced; if the oral data matches the data to be spliced, splice the data to be spliced ​​with the oral data so that the data to be spliced ​​is aligned with the oral data.

[0141] Optionally, the gingival alignment unit 400 is also configured to: trigger the head-mounted camera to acquire new oral data from multiple regions if the oral data does not match the data to be stitched; the new oral data is used to re-match with the data to be stitched.

[0142] Optionally, the data to be spliced ​​includes gingival model data, which is used to characterize the corresponding gingival model.

[0143] Optionally, the oral data may also include pre-added spot features on the region, which may include optical features projected onto the region using third-party optical equipment.

[0144] The dot display unit 500 is configured to display matched dots and unmatched dots in the head-mounted display; wherein the display effect of matched dots is different from that of unmatched dots.

[0145] Each of the units described above uses a head-mounted display to replace the extraoral scanner to obtain coded scanning rod frame point data. The first stitched data is generated using feature points in the coded scanning rod frame point data that match the standard data to help track the scanning progress of the coded scanning rod. Furthermore, adjustment suggestions are generated using unmatched feature points to help dentists detect whether any coded scanning rods have been missed, thereby reducing the operational threshold of the impression taking process.

[0146] This application also provides a computer-readable storage medium including a stored program, wherein the program executes the head-mounted display-based digital modeling method provided in this application.

[0147] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0148] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application. Industrial applicability

[0149] The technical solution provided in this application stitches and matches multi-frame feature point data acquired in real time by a head-mounted display camera to obtain coded scanning rod frame point data. Based on the feature points in the coded scanning rod frame point data that match standard data, the corresponding standard data is stitched together in the head-mounted display to generate first stitched data. When the number of scanning rods in the first stitched data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data. This application uses a head-mounted display to replace an extraoral scanner to obtain coded scanning rod frame point data, uses the feature points in the coded scanning rod frame point data that match standard data to generate first stitched data to assist in tracking the scanning progress of the coded scanning rods, and uses the unmatched feature points to generate adjustment suggestions to assist dentists in detecting whether any coded scanning rods have been missed, thereby reducing the operational threshold of the impression taking process.

Claims

1. A digital modeling method based on a head-mounted display, comprising: The multi-frame feature point data collected in real time by the head-mounted display camera is stitched together and matched to obtain the coded scanning rod frame point data. Based on the feature points in the frame point data of the coded scanning rod that match the standard data, the corresponding standard data is spliced ​​in the head-mounted display to generate the first spliced ​​data; When the number of scanning rods in the first splicing data does not match the number of scanning rods installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data.

2. The method according to claim 1, wherein, The method further includes: Based on the oral cavity scan data collected by the intraoral scanner, the data to be stitched is obtained; Obtain oral cavity data from multiple regions captured by the head-mounted display camera; the oral cavity data includes feature point data of coded scanning rods within the regions, and the oral cavity data is image data and / or model data; The data to be spliced ​​is spliced ​​with multiple oral cavity data respectively, so that the data to be spliced ​​is aligned with the multiple oral cavity data; Feature point data from multiple oral cavity data are concatenated with the frame point data of the coded scanning rod, so that the data to be concatenated is aligned with the frame point data of the coded scanning rod.

3. The method according to claim 2, wherein, The data to be spliced ​​is spliced ​​with multiple oral cavity data respectively, so that the data to be spliced ​​is aligned with the multiple oral cavity data, including: For each oral cavity data, determine whether the oral cavity data matches the data to be spliced; If the oral cavity data matches the data to be spliced, the data to be spliced ​​and the oral cavity data are spliced ​​together so that the data to be spliced ​​and the oral cavity data are aligned.

4. The method according to claim 3, wherein, The method further includes: If the oral cavity data does not match the data to be stitched together, the head-mounted display camera is triggered to collect new oral cavity data from multiple regions; the new oral cavity data is used to re-match with the data to be stitched together.

5. The method according to any one of claims 1-4, wherein, Based on feature points in the coded scanning rod frame point data that match the standard data, the corresponding standard data is stitched together in the head-mounted display to generate the first stitched data, including: When feature point data for each frame is obtained, the target encoded scanning rod that matches the feature points in the feature point data of historical frames is determined. For each feature point in the feature point data of each frame, if the feature point belongs to the target encoding scan bar, the feature point is marked as a matched point; if the feature point does not belong to the target encoding scan bar, the feature point is marked as an unmatched point. The unmatched points are matched with the feature points of the coded scanning rods pre-stored in the scanning rod database to obtain the corresponding matching results; If the matching result is a successful match, the unmatched point will be remarked as a matched point; Based on the matched points in the feature point data of each frame and the feature points in the feature point data of the historical frames, the feature points that match the standard data in the coded scan rod frame point data are determined. Based on the feature points in the frame point data of the coded scanning rod that match the standard data, the corresponding standard data is spliced ​​in the head-mounted display to generate the first spliced ​​data.

6. The method according to any one of claims 1-5, wherein, When the number of scanning rods in the first stitched data does not match the number of scanning rods to be installed, corresponding adjustment suggestions are generated based on the unmatched feature points in the coded scanning rod frame point data, including: When the number of scanning rods in the first stitched data does not match the number of scanning rods installed, the attribute information of the unmatched feature points in the coded scanning rod frame point data is determined; the attribute information includes at least position, distance, sharpness, and frequency of occurrence. Based on the attribute information of the unmatched feature points, corresponding adjustment suggestions are generated; The adjustment suggestions include adjusting the patient's mouth posture, adjusting the spacing between the coded scanning rods, prompting the user to re-download the scanning rod database, and / or prompting the user to adjust the distance between the head-mounted display and the coded scanning rods.

7. The method according to claim 5, wherein, The method further includes: The matched points and the unmatched points are displayed in the head-mounted display; wherein the display effect of the matched points is different from that of the unmatched points.

8. The method according to any one of claims 1-7, wherein, The standard data includes standard scan bar data, which is used to identify the relative positional relationship between multiple coded scan bars.

9. The method according to any one of claims 1-8, wherein, The adjustment suggestion is used to prompt the head-mounted display wearer to perform the corresponding adjustment action so that the coded scanning rod frame point data adds feature points that match the missing data; the missing data includes the standard data portion corresponding to the missing coded scanning rod.

10. The method according to claim 2, wherein, The data to be spliced ​​includes gingival model data, which is used to characterize the corresponding gingival model.

11. The method according to claim 2, wherein, The oral cavity data also includes pre-added spot features on the region, which include optical features projected onto the region using a third-party optical device.

12. The method of any one of claims 1-11, wherein, The first spliced ​​data is used to determine the relative positional relationship between each of the coded scanning rods.

13. A head-mounted display, comprising: The data determination unit is configured to stitch together and match multi-frame feature point data acquired in real time by the head-mounted display camera to obtain coded scanning rod frame point data. The data stitching unit is configured to stitch the corresponding standard data in the head-mounted display based on the feature points in the frame point data of the coded scanning rod that match the standard data, thereby generating the first stitched data; The suggestion generation unit is configured to generate corresponding adjustment suggestions based on the unmatched feature points in the coded scan rod frame point data when the number of scan rods in the first stitched data does not match the number of scan rods installed.

14. A head-mounted display-based digital impression system, comprising an intraoral scanner and a head-mounted display, wherein the intraoral scanner is configured to acquire oral cavity scan data of a patient, and the head-mounted display is configured to perform the head-mounted display-based digital impression method according to any one of claims 1-12.

15. A storage medium comprising a stored program, wherein, The program is executed by the processor to perform the digital modeling method based on the head-mounted display as described in any one of claims 1-12.

Images