Model control method and system based on augmented reality, and head-mounted display

Abstract

Disclosed in the present application are a model control method and system based on augmented reality, and a head-mounted display. The method comprises: on the basis of an augmented reality environment of a head-mounted display, generating a tooth model corresponding to patient gingival data that is collected by an intraoral scanning device; and when the augmented reality environment enters a model editing interface, on the basis of motion-sensing interaction information collected by the head-mounted display, triggering intraoral scanning software in the head-mounted display to execute a corresponding control instruction, such that the tooth model makes a change matching the control instruction. By means of the control instruction, the editing and control for the tooth model can be realized, such that a doctor can see an intuitive and accurate three-dimensional model in an avatar, and also edits the tooth model conveniently and quickly by means of a motion-sensing interaction behavior, and thus a conventional mouse+keyboard+screen mode is discarded, and the editing and control for the tooth model are completed by means of augmented reality.

Description

Augmented reality-based model control methods, systems, and head-mounted displays Cross-reference to related applications

[0001] This application claims priority to Chinese Patent Application No. 2025100966157, filed on January 21, 2025, entitled "Model Control Method, System and Head-Mounted Display Based on Augmented Reality", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of oral healthcare, and more particularly to an augmented reality-based model control method, system, and head-mounted display. Background Technology

[0003] In the current scenario for viewing 3D models in the field of oral healthcare, a dental scanner is typically used as a peripheral device for a computer. Corresponding software drives and controls the scanner. After scanning, the model can be viewed within the computer software or uploaded to the cloud for viewing on a webpage. Generally, the scenarios for viewing and editing 3D models in oral healthcare are: the 3D model (e.g., a tooth model) is displayed on a monitor, and the model is manipulated using a mouse and keyboard; or the model is displayed on a webpage, and the user can access the webpage on a mobile phone or tablet and manipulate the 3D model by touching the screen.

[0004] Existing model control methods typically involve a computer plus a screen, mobile phone, or tablet, requiring a mouse, keyboard, touchscreen, or touchpad to control the model's display and editing. Furthermore, displaying a 3D model on a screen is not intuitive. Secondly, using physical peripherals to operate the model is inconvenient in medical procedures. Doctors must maintain a sterile environment by removing their gloves, and then putting them back on afterward, a time-consuming process that results in a poor user experience. Additionally, displaying a 3D model on a screen requires a facial scanner to scan the patient's face, creating a 3D model that demands spatial reasoning skills and presents communication barriers, leading to low efficiency. Summary of the Invention

[0005] This application provides an augmented reality-based model control method and head-mounted display, which is configured to, on the premise of acquiring three-dimensional data (i.e., three-dimensional model) inside the patient's oral cavity, abandon the traditional mouse, keyboard and screen method and use augmented reality to complete the editing control and interactive display of the three-dimensional model.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] An augmented reality-based model control method includes: generating a tooth model corresponding to patient gingival data collected by an intraoral scanning device based on an augmented reality environment of a head-mounted display; when the augmented reality environment enters the model editing interface, triggering the intraoral scanning software within the head-mounted display to execute corresponding control commands based on the haptic interaction information collected by the head-mounted display, so that the tooth model undergoes changes matching the control commands; wherein, the haptic interaction information includes one or more of the following: doctor's gestures, intraoral scanning device handheld posture, head posture, and head-mounted display displacement trajectory; the triggering condition for the control commands is: the haptic interaction information is based on the target gesture or the target intraoral scanning device handheld posture as the initial behavior; the intraoral scanning software is system software that supports editing the tooth model.

[0008] Optionally, if the somatosensory interaction information meets the first behavioral condition, the oral scanning software in the head-mounted display is triggered to execute the first control command; wherein, the first behavioral condition includes at least one of the following: the doctor's gesture is waving; or the hand position of the oral scanning device is rotating the oral scanning device; or the head position is rotating the head; or the head-mounted display displacement trajectory is a circumferential trajectory; the first control command is configured to trigger the tooth model to rotate.

[0009] Optionally, if the somatosensory interaction information meets the second behavioral condition, the oral scanning software in the head-mounted display is triggered to execute the second control command; wherein, the second behavioral condition includes at least one of the following: the doctor's hand gesture is an open palm with five fingers together and the palm facing forward; or the oral scanning device is held vertically and moved horizontally after pressing the first preset button.

[0010] Alternatively, the head-mounted display displacement trajectory may be a translation trajectory; the second control command is configured to trigger the tooth model to translate.

[0011] Optionally, if the haptic interaction information meets the third behavioral condition, the intraoral scanning software within the head-mounted display is triggered to execute a third control command; wherein the third behavioral condition includes at least one of the following: the doctor's gesture is either fingers opening and closing or fingers closing; or the intraoral scanning device is held horizontally and moved vertically after pressing the second preset button; or the head-mounted display is displaced to change the distance between itself and the specified target; the third control command is configured to trigger scaling of the tooth model; the scaling type includes shrinking and enlarging; when the doctor's gesture is fingers opening and closing, or the vertical movement direction of the intraoral scanning device is upward, or the distance between the head-mounted display and the specified target is decreasing, the tooth model is enlarged; when the doctor's gesture is fingers closing, or the vertical movement direction of the intraoral scanning device is downward, or the distance between the head-mounted display and the specified target is increasing, the tooth model is shrunk.

[0012] Optionally, when the dental model is scaled by moving vertically after the horizontally placed oral scanning device is pressed, a scaling scale is displayed in the side view of the head-mounted display, and the scaling scale is locked when the second preset button is released; after the second preset button is pressed and released repeatedly, the scaling scale reaches the limit value.

[0013] Optionally, if the haptic interaction information meets the fourth behavior condition, the intraoral scanning software in the head-mounted display is triggered to execute the fourth control command; wherein, the fourth behavior condition includes at least one of the following: the doctor's hand gesture is a fist with the palm facing up; or the intraoral scanning device is held in a vertical position after shaking it up and down a preset number of times and then continuously clicking the third preset button; the fourth control command is configured to trigger the tooth model to switch to the jaw open state.

[0014] Optionally, if the haptic interaction information meets the fifth behavior condition, the intraoral scanning software in the head-mounted display is triggered to execute the fifth control command; wherein, the fifth behavior condition includes at least one of the following: the doctor's hand gesture is an open palm facing upwards; or the intraoral scanning device is held in a vertical position after shaking up and down a preset number of times and then continuously clicking the fourth preset button; the fifth control command is configured to trigger the tooth model to switch to the jaw closure state.

[0015] Optionally, the augmented reality environment also includes a finger model and an oral scanning device model. The finger model is determined based on the doctor's hand posture captured by the head-mounted display, and the oral scanning device model is determined based on the oral scanning device posture captured by the head-mounted display. When the model editing interface is the tooth position marking function interface, if the head-mounted display's viewing angle meets the requirements, the tooth position label is attached to the position corresponding to the fingertip of the finger model or the top of the oral scanning device model. If the head-mounted display's viewing angle does not meet the requirements, the tooth position label is attached to the position corresponding to the extension line of the finger model or the extension line of the oral scanning device model.

[0016] Optionally, when the model editing interface is the hole-cutting editing interface or the brush painting interface, if the haptic interaction information meets the sixth behavior condition, the oral scanning software in the head-mounted display is triggered to execute the sixth control command; wherein, the sixth behavior condition is: the doctor's gesture is a circle made of thumb and index finger; the sixth control command is configured to trigger the tooth model to revert to the previous operation state.

[0017] Optionally, when the model editing interface is the hole-cutting editing interface or the brush painting interface, if the haptic interaction information meets the seventh behavior condition, the oral scanning software in the head-mounted display is triggered to execute the seventh control command; wherein, the seventh behavior condition is: the doctor's gesture is a thumb and index finger check; the seventh control command is configured to trigger the tooth model to advance to the next operation state.

[0018] Optionally, when the model editing interface is a custom rotation center interface, the augmented reality environment generates the rotation center of the tooth model, triggers the selection of the rotation center based on the alignment direction of the finger model or the oral scanning device model, and determines the position of the rotation center based on the drag event of the finger model or the oral scanning device model.

[0019] A head-mounted display includes: a model generation unit configured to generate a tooth model corresponding to patient gingival data collected by an intraoral scanning device based on the augmented reality environment of the head-mounted display; and an instruction execution unit configured to, when the augmented reality environment enters the model editing interface, trigger the intraoral scanning software within the head-mounted display to execute corresponding control instructions based on the haptic interaction information collected by the head-mounted display, so that the tooth model undergoes changes matching the control instructions; wherein, the haptic interaction information includes one or more of the following: doctor's gestures, intraoral scanning device handheld posture, head posture, and head-mounted display displacement trajectory; the triggering condition for the control instructions is: the haptic interaction information is based on the target gesture or the target intraoral scanning device handheld posture as the initial behavior; and the intraoral scanning software is system software that supports editing the tooth model.

[0020] An augmented reality-based model control system includes a head-mounted display (HUD) and an oral scanning device; wherein the HUD is configured to execute an augmented reality-based model control method; and the oral scanning device is configured to acquire patient gingival data and send the patient gingival data to the HUD.

[0021] The technical solution provided in this application generates a tooth model corresponding to the patient's gingival data collected by an intraoral scanning device, based on the augmented reality environment of a head-mounted display. When the augmented reality environment enters the model editing interface, the intraoral scanning software within the head-mounted display is triggered to execute corresponding control commands based on the haptic interaction information collected by the head-mounted display, causing the tooth model to change in accordance with the control commands. This application utilizes the haptic interaction information collected by the head-mounted display to control the intraoral scanning software within the head-mounted display to execute corresponding control commands, thereby realizing the editing control of the tooth model. This allows doctors to see an intuitive and accurate 3D model in their head, and simultaneously use haptic interaction to conveniently and quickly edit the tooth model, thus abandoning the traditional mouse, keyboard, and screen method and using augmented reality to complete the editing control of the tooth model. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 is a flowchart illustrating a model control method based on augmented reality provided in an embodiment of this application;

[0024] Figure 2 is a schematic diagram of a jaw opening state provided in an embodiment of this application;

[0025] Figure 3 is a schematic diagram of the architecture of a head-mounted display provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0028] Figure 1 shows a flowchart of an augmented reality-based model control method provided in an embodiment of this application, which includes the following steps.

[0029] S101: Based on the augmented reality environment of the head-mounted display, generate a tooth model corresponding to the patient's gingival data collected by the intraoral scanning device.

[0030] The headset wearers include at least doctors. If the doctor is communicating with the patient, the patient can wear another headset. This other headset, also based on the local augmented reality environment, generates a tooth model corresponding to the patient's gingival data collected by the intraoral scanner. Additionally, the doctor's headset can use its camera to capture facial images of the patient and send these images to another headset worn by the patient. This allows the other headset to overlay the facial images onto the tooth model (meaning the teeth on the model are aligned with the teeth shown in the facial image).

[0031] In some examples, the types of head-mounted displays include, but are not limited to, head-mounted displays (such as augmented reality devices) and smart glasses. Generally speaking, the head-mounted display and its operating system provide the basic hardware and computing resources required for 3D modeling, including but not limited to cameras (such as binocular cameras, cameras), microphones (such as microphones), etc.

[0032] It should be emphasized that, for the editing and control of dental models, the head-mounted display shown in this application embodiment has at least the following capabilities: 1. The head-mounted display can install software like a computer and supports custom peripherals. For example, the head-mounted display (HMD) can install mouth scanning software, with the mouth scanning device acting as an external connection. The HMD will transmit the received haptic interaction information to the mouth scanning software installed within the HMD; 2. The HMD has accurate and powerful gesture recognition capabilities. With proper configuration of the gesture library or mouth scanning device posture library, it can recognize corresponding gestures and postures; 3. The HMD supports custom gestures. When custom mode is enabled, it provides information within the captured hand area to the mouth scanning software. The top part of the mouth scanning device (e.g., the scanning head) can be used to replace fingers to touch virtual buttons (for prosthetics), or the mouth scanning device can be held to perform gesture operations; 4. The HMD has spatial computing capabilities, distinguishing between quick gestures and touching virtual buttons in the augmented reality environment; 5. The HMD can simultaneously display real-world and augmented reality images; 6. The HMD has a built-in gyroscope, converting head rotation into model or image movement information; 7. The HMD has computing power, supporting 3D model editing, stitching, and conversion calculations without image lag; 8. The HMD has indoor positioning and tracking capabilities, determining its own displacement trajectory.

[0033] S102: When the augmented reality environment enters the model editing interface, based on the haptic interaction information collected by the head-mounted display, the oral scanning software in the head-mounted display is triggered to execute the corresponding control commands, so that the tooth model undergoes changes that match the control commands.

[0034] The haptic interaction information includes one or more of the following: doctor's gestures, the hand position of the intraoral scanning device, head posture, and head-mounted display displacement trajectory. The trigger condition for control commands is that the haptic interaction information is based on the target gesture or the target hand position of the intraoral scanning device as the initial action. The intraoral scanning software is system software that supports editing dental models.

[0035] In some examples, the model editing interface is an interactive screen provided by the oral scanning software. When the augmented reality environment of the head-mounted display enters the model editing interface, it can be assumed that the doctor is editing the dental model. At this time, the dental model is a virtual 3D stereoscopic display. In the upper left corner of the model editing interface, there is a virtual button configured to close the interface. In the lower right corner and the lower edge area of ​​the model editing interface, there are corresponding virtual buttons configured to enable model editing.

[0036] Generally speaking, the gestures of doctors and the hand positions of intraoral scanning devices in motion-sensing interaction information are similar to flag signals, using limited props combined with different postures to express different meanings. For the embodiments of this application, the motion-sensing interaction information that triggers the execution of control commands must start with the target gesture or the target hand position of the intraoral scanning device. This starting behavior is usually an err gesture (i.e., actively not recognizing gestures). The err gesture can be used to indicate the end of the previous gesture and to recognize a new gesture.

[0037] In a possible implementation, the target gesture can be a left-right hand shaking, and the target mouth scanner hand-holding posture can be a left-right hand shaking of the mouth scanner.

[0038] Optionally, if the somatosensory interaction information meets the first behavioral condition, the oral scanning software in the head-mounted display is triggered to execute the first control command; wherein, the first behavioral condition includes at least one of the following: the doctor's gesture is waving, or the hand position of the oral scanning device is rotating the oral scanning device, or the head position is rotating the head, or the displacement trajectory of the head-mounted display is a circumferential trajectory; the first control command is configured to trigger the tooth model to rotate.

[0039] In some examples, the tooth model is rotated by default to rotating the model center, which can be achieved by rotating the center of the bounding box of the tooth model (i.e., the spatial geometric center of the smallest cuboid that can enclose the tooth model in the augmented reality environment).

[0040] In one possible implementation, the tooth model is rotated by turning the head. This requires using the head-mounted display's gyroscope information to determine the head posture in real time. Therefore, the "fixed position rotation" function of the head-mounted display needs to be enabled beforehand in the display's interface. With "fixed position rotation" enabled, the display area remains directly in front of the eyes, and the tooth model remains at the center of the display's field of view. When the head rotates, the intraoral scanning software within the head-mounted display executes the first control command, causing the tooth model to rotate. Specifically, turning the head to the right causes the tooth model to rotate to the left, turning the head to the left causes it to rotate to the right, turning the head upwards (tilting the head up) causes it to rotate downwards, and turning the head downwards (tilting the head down) causes it to rotate upwards. It is important to note that when collecting and recognizing head posture, the head-mounted display processes subtle non-user-subjective movements to prevent the display from becoming overly sensitive and causing 3D motion sickness in the user.

[0041] In one possible implementation, the dental model is controlled to rotate by rotating the oral scanning device, and the rotation direction of the oral scanning device is consistent with the rotation direction of the dental model.

[0042] In a possible implementation, the dental model is rotated by waving (requiring a rapid wave). Specifically, the head-mounted display can collect and identify the start and end positions, speed, and angle of the hand movements to determine whether the doctor's gesture is a wave. The start and end positions include whether they are near the position displayed on the dental model, while the speed and angle can be analyzed by the AI ​​in the head-mounted display.

[0043] In a possible implementation, the tooth model is rotated by turning the head-mounted display (or by the doctor). This requires real-time head posture determination using the head-mounted display's positioning information. Therefore, the "Fixed Position SLAM" function of the head-mounted display needs to be enabled beforehand in its interface. In reality, the tooth model remains stationary during SLAM rotation; it can be understood that the person, or the head-mounted display, is moving. When switched to SLAM mode, the tooth model remains fixed within the head-mounted display's field of view. The general logic is that when the tooth model is first displayed, the head-mounted display combines the tooth model with the background to form the initial position for SLAM judgment. As the doctor moves around the tooth model, the SLAM positioning is used to generate changes in the model's perspective corresponding to the environment. Specifically, controlling the rotation of the tooth model based on the walking trajectory is essentially an effect achieved by rotating the head-mounted display's perspective around the tooth model.

[0044] The principle behind SLAM (Simultaneous Localization and Mapping) is a conventional robot autonomous localization and navigation technology, which will not be elaborated here.

[0045] Optionally, if the haptic interaction information meets the second behavioral condition, the intraoral scanning software in the head-mounted display is triggered to execute the second control command; wherein, the second behavioral condition includes at least one of the following: the doctor's hand gesture is an open palm with five fingers together and palm facing forward, or the intraoral scanning device is held vertically after pressing the first preset button and then moved horizontally, or the head-mounted display displacement trajectory is a translation trajectory; the second control command is configured to trigger the dental model to translate.

[0046] In one possible implementation, the dental model is translated by opening the palm, keeping the fingers together, and facing forward. This gesture is similar to wiping a glass, and the direction of wiping the glass can be regarded as the direction of translation of the dental model.

[0047] In one possible implementation, after pressing a first preset button on the intraoral scanner, the device is then held vertically and moved horizontally to control the translation of the dental model. Because the response of the first preset button and the posture of the intraoral scanner are taken into account, the reliability of this control command is higher. Specifically, the direction of vertical translation of the intraoral scanner can be considered the translation direction of the dental model.

[0048] In one possible implementation, the dental model is translated by moving the head-mounted display (or the doctor moving it). After the head-mounted display moves relative to the indoor environment, the dental model moves to the same display area. However, if the dental model is large and the augmented reality environment space is small, the SLAM operation space will be relatively small.

[0049] Optionally, if the haptic interaction information meets the third behavior condition, the intraoral scanning software within the head-mounted display is triggered to execute a third control command. The third behavior condition is: the doctor's gesture is either fingers opening and closing or fingers closing; or the intraoral scanning device is held horizontally and moved vertically after pressing the second preset button; or the head is displaced to change the distance to the specified target. The third control command is configured to trigger scaling of the tooth model. The scaling type includes shrinking and enlarging. When the doctor's gesture is fingers opening and closing, or the vertical movement direction of the intraoral scanning device is upward, or the distance between the head-mounted display and the specified target is decreasing, the tooth model is enlarged. When the doctor's gesture is fingers closing, or the vertical movement direction of the intraoral scanning device is downward, or the distance between the head-mounted display and the specified target is increasing, the tooth model is shrunk.

[0050] In a possible implementation, the dental model is controlled to shrink by bringing the fingers together, and to enlarge by opening the fingers. When the doctor's hand is in a fist, the dental model is controlled to maintain the current scaling state.

[0051] Optionally, when the dental model is scaled by moving vertically after the horizontally placed oral scanning device is pressed, a scaling scale is displayed in the side view of the head-mounted display, and the scaling scale is locked when the second preset button is released; after the second preset button is pressed and released repeatedly, the scaling scale reaches the limit value.

[0052] In one possible implementation, after the second preset button is pressed and released a specified number of times, the zoom level reaches its limit, at which point the zoom scale can no longer update. Alternatively, the zoom level can be restored to the specified zoom level by clicking the zoom scale.

[0053] In a possible implementation, the head-mounted display is moved to change the distance between itself and a designated target (i.e., the doctor moves to change the distance between himself and the designated target, which can be understood as the patient), and the tooth model is controlled to scale. Essentially, this is the effect achieved by moving the head-mounted display's viewpoint and the distance between itself and the tooth model. When the head-mounted display is closer to the tooth model, the tooth model will be magnified; when the head-mounted display is farther away from the tooth model, the tooth model will be shrunk.

[0054] Optionally, if the haptic interaction information meets the fourth behavior condition, the intraoral scanning software in the head-mounted display is triggered to execute the fourth control command; wherein, the fourth behavior condition includes at least one of the following: the doctor's hand gesture is a fist with the palm facing up, or the intraoral scanning device is held in a vertical position and shaken up and down a preset number of times before continuously clicking the third preset button; the fourth control command is configured to trigger the tooth model to switch to the jaw open state.

[0055] In some examples, the tooth model can be switched to an open jaw state by shaking it up and down three times, followed by double-clicking the third preset button on the intraoral scanner vertically. "Vertical" here refers to holding the intraoral scanner in a vertical position.

[0056] In a possible implementation, the tooth model can also be controlled to open by triggering a virtual button on the model editing interface. Specifically, clicking the virtual button to open the jaw can trigger the tooth model to switch to the jaw-open state.

[0057] In some examples, the tooth model is placed in an open jaw position, as shown in Figure 2.

[0058] Optionally, if the haptic interaction information meets the fifth behavior condition, the oral scanning software in the head-mounted display is triggered to execute the fifth control command; wherein, the fifth behavior condition includes at least one of the following: the doctor's hand gesture is an open palm facing upwards, or the oral scanning device is held in a vertical position after shaking it up and down a preset number of times and then continuously clicking the fourth preset button; the fifth control command is configured to trigger the tooth model to switch to the jaw closure state.

[0059] In some examples, the tooth model can be switched to a closed jaw state by shaking it up and down three times and then double-clicking the fourth preset button on the oral scanner vertically.

[0060] In a possible implementation, the tooth model can also be controlled to close by triggering a virtual button on the model editing interface. Specifically, clicking the virtual button to close the jaw can trigger the tooth model to switch to a closed jaw state.

[0061] Optionally, the augmented reality environment also includes a finger model and an oral scanning device model. The finger model is determined based on the doctor's hand posture captured by the head-mounted display, and the oral scanning device model is determined based on the oral scanning device posture captured by the head-mounted display. When the model editing interface is the tooth position marking function interface, if the head-mounted display's viewing angle meets the requirements, the tooth position label is attached to the position corresponding to the fingertip of the finger model or the top of the oral scanning device model. If the head-mounted display's viewing angle does not meet the requirements, the tooth position label is attached to the position corresponding to the extension line of the finger model or the extension line of the oral scanning device model.

[0062] In a possible implementation, when the model editing interface is the tooth position marking function interface, the tooth model is displayed as a full-mouth dentition by default. The right side of the head-mounted display's field of view shows a full-mouth marked tooth position map, and the corresponding tooth position can be selected via a virtual button on the tooth position marking function interface. After selecting a tooth position via the virtual button, the augmented reality environment generates a corresponding tooth position label and determines whether the head-mounted display's viewing angle meets the requirements. Specifically, if the distance between the head-mounted display's viewing angle and the tooth model is less than a preset threshold, the head-mounted display's viewing angle is determined to meet the requirements; if the distance is greater than or equal to the preset threshold, the head-mounted display's viewing angle is determined to not meet the requirements. After selecting a tooth position on the tooth model using the hand model and the intraoral scanning device model, the tooth position label will be simultaneously attached to the corresponding selected position.

[0063] Optionally, when the model editing interface is the hole-cutting editing interface or the brush painting interface, if the haptic interaction information meets the sixth behavior condition, the oral scanning software in the head-mounted display is triggered to execute the sixth control command; wherein, the sixth behavior condition is: the doctor's gesture is a circle made of thumb and index finger; the sixth control command is configured to trigger the tooth model to revert to the previous operation state.

[0064] In a possible implementation, when the model editing interface is the hole-cutting editing function interface, the tooth model is displayed as a full-mouth dentition by default. The corresponding tooth position can be selected using a virtual button on the hole-cutting editing function interface. After selecting a tooth position using the virtual button, the augmented reality environment generates a corresponding hole label. The system also determines whether the head-mounted display's viewing angle meets the requirements. Specifically, if the distance between the head-mounted display's viewing angle and the tooth model is less than a preset threshold, the viewing angle is considered acceptable; if the distance is greater than or equal to the preset threshold, the viewing angle is considered unacceptable. After selecting a tooth position on the tooth model using the hand model and the intraoral scanning device model, a hole label will be simultaneously attached to the selected position.

[0065] Understandably, after each successful hole-digging, the tooth model will update its corresponding state. By circling the tooth model with the thumb and forefinger, you can control the tooth model to revert to the previous operation state. For example, if the tooth model in the current operation state has two holes and the tooth model in the previous operation state has one hole, you can reset the second hole.

[0066] In a possible implementation, when the model editing interface is a brush smearing function interface, the brush smearing function interface is provided with virtual buttons corresponding to drawing, brushing, painting, and smearing. By selecting a virtual button, the augmented reality environment will generate a corresponding tool label. If the head-mounted display's viewing angle meets the requirements, the tool label is attached to the position corresponding to the fingertip of the finger model or the top of the mouth scanning device model. If the head-mounted display's viewing angle does not meet the requirements, the tool label is attached to the position corresponding to the extension line of the finger model or the extension line of the mouth scanning device model.

[0067] Understandably, after each successful operation, the tooth model will update its corresponding state. By circling the tooth model with your thumb and forefinger, you can control the tooth model to revert to the previous operation state. Suppose that the tooth model in the current operation state has two smear marks, and the tooth model in the previous operation state has one smear mark, so that the second smear mark can be reset.

[0068] Optionally, when the model editing interface is the hole-cutting editing interface or the brush painting interface, if the haptic interaction information meets the seventh behavior condition, the oral scanning software in the head-mounted display is triggered to execute the seventh control command; wherein, the seventh behavior condition is: the doctor's gesture is a thumb and index finger check; the seventh control command is configured to trigger the tooth model to advance to the next operation state.

[0069] Understandably, by circling the thumb and forefinger to control the tooth model to revert to the previous few operation states, one can also circling the thumb and forefinger to control the tooth model to advance to the latest operation state, so as to review the operation state of the tooth model corresponding to each editing operation (i.e., the hole-cutting operation or the brush painting operation).

[0070] In some examples, assuming the doctor, after obtaining a closed upper and lower jaw, needs to mark the position of tooth 14 in the upper jaw and then remove a corner of the corresponding position in the lower jaw, the specific sequence of the doctor's gestures is as follows: 1. Control the finger model to click the virtual button for tooth marking; 2. Control the finger model to select the virtual button for the tooth position number; 3. Control the finger model to attach the tooth position label attached to the fingertip to the corresponding position 14 in the tooth model, and double-click to select position 14; 4. Shake the hand left and right, while switching to palm up, wait for 1 second, then turn the palm up, then clench the fist, triggering the tooth model to switch to an open jaw state; 5. Shake the hand left and right, with the palm open and facing forward, similar to wiping a window, to translate the tooth model to the appropriate position; 6. Wave the hand to trigger the tooth model to rotate to the appropriate position; 7. Control the finger model to click the virtual button for hole editing; 8. Double-click the corresponding position on the tooth model to make a hole.

[0071] In some examples, assuming the dentist obtains a closed maxillary and mandibular jaw, and then marks the position of tooth 14 in the maxilla, and then removes a corner of the corresponding position of tooth 14 in the mandible, the specific operating sequence of the intraoral scanner holding posture is as follows: 1. Control the intraoral scanner model to click the tooth position marking virtual button; 2. Control the intraoral scanner model to select the tooth position number virtual button; 3. Control the intraoral scanner model to attach the tooth position label attached to the top part to the corresponding tooth position 14 on the tooth model, and double-click to select tooth position 14; 4. Shake the intraoral scanner left and right, control the intraoral scanner model to shake up and down three times, and then double-click the fourth preset button vertically to switch the tooth model to the closed jaw state; 5. Shake the intraoral scanner left and right, press the first preset button, and vertically hold the intraoral scanner to move it horizontally to move the tooth model to the appropriate position; 6. Rotate the intraoral scanner to rotate the tooth model to the appropriate position; 7. Control the intraoral scanner model to click the hole editing virtual button; 8. Double-click the corresponding position on the tooth model to make a hole.

[0072] Optionally, when the model editing interface is a custom rotation center interface, the augmented reality environment generates the rotation center of the tooth model, triggers the selection of the rotation center based on the alignment direction of the finger model or the oral scanning device model, and determines the position of the rotation center based on the drag event of the finger model or the oral scanning device model.

[0073] In a possible implementation, a virtual "Custom Rotation Center" button is provided within the head-mounted display's field of view. Clicking the "Custom Rotation Center" button switches the model editing interface to the custom rotation center interface, and a semi-transparent spherical suction area appears in the augmented reality environment. This custom rotation center interface can inform the doctor through information display that the center of the transparent sphere is a movable rotation center, at which point the tooth model is displayed semi-transparently. Extending a finger (usually the index or middle finger) or using the scanning head of an intraoral scanner aligns with the rotation center to control the alignment direction of the finger model or the intraoral scanner model. If the head-mounted display's view does not meet the requirements, "Alignment" indicates that the extended ray of the finger or scanning head is aligned. The software recognizes the hand + finger extension or the intraoral scanner head to trigger selection. At this time, the semi-transparent sphere corresponding to the rotation center is highlighted to indicate selection. Moving the finger model or the intraoral scanner model and dragging it to the designated position, or making a fist gesture to abandon the current gesture, or double-clicking the intraoral scanner button, the rotation center remains in the final position. Pressing and holding the button on the intraoral scanner or touching the "OK" button at the bottom of the field of view confirms the modification of the rotation center.

[0074] In some examples, the head-mounted display also features a "Restore Default Rotation Center" virtual button and a "Cancel" virtual button within its field of view. The "Restore Default Rotation Center" virtual button is configured to trigger the tooth model's rotation center to return to its initial state, while the "Cancel" virtual button is configured to end the modification of the rotation center.

[0075] The process described in S101-S102 above utilizes the haptic interaction information collected by the head-mounted display to control the intraoral scanning software within the head-mounted display to execute corresponding control commands, thereby enabling the editing and control of the dental model. This allows doctors to see an intuitive and accurate 3D model in their head and conveniently and quickly edit the dental model using haptic interaction, thus abandoning the traditional mouse, keyboard and screen method and using augmented reality to complete the editing and control of the dental model.

[0076] Figure 3 shows a schematic diagram of the architecture of a head-mounted display provided in an embodiment of this application, including the following units.

[0077] The model generation unit 100 is configured to generate a tooth model corresponding to the patient's gingival data collected by the intraoral scanning device in an augmented reality environment based on the head-mounted display.

[0078] The instruction execution unit 200 is configured to, when the augmented reality environment enters the model editing interface, trigger the intraoral scanning software within the head-mounted display to execute corresponding control instructions based on the haptic interaction information collected by the head-mounted display, so that the dental model undergoes changes that match the control instructions; wherein, the haptic interaction information includes one or more of the following: doctor's gestures, intraoral scanning device handheld posture, head posture, and head-mounted display displacement trajectory; the triggering condition for the control instructions is: the haptic interaction information is based on the target gesture or the target intraoral scanning device handheld posture as the starting behavior; the intraoral scanning software is system software that supports editing the dental model.

[0079] Optionally, the instruction execution unit 200 is specifically configured to: if the somatosensory interaction information meets the first behavioral condition, trigger the oral scanning software in the head-mounted display to execute the first control instruction; wherein the first behavioral condition includes at least one of the following: the doctor's gesture is waving; or the hand position of the oral scanning device is rotating the oral scanning device; or the head position is rotating the head; or the head-mounted display displacement trajectory is a circumferential trajectory; the first control instruction is configured to trigger the tooth model to rotate.

[0080] Optionally, the instruction execution unit 200 is specifically configured to: if the somatosensory interaction information meets the second behavioral condition, trigger the intraoral scanning software in the head-mounted display to execute the second control instruction; wherein the second behavioral condition includes at least one of the following: the doctor's hand gesture is an open palm with five fingers together and the palm facing forward; or the intraoral scanning device is held vertically after pressing the first preset button and then moved; or the head-mounted display displacement trajectory is a translation trajectory; the second control instruction is configured to trigger the dental model to translate.

[0081] Optionally, the instruction execution unit 200 is specifically configured to: if the haptic interaction information meets the third behavioral condition, trigger the oral scanning software in the head-mounted display to execute a third control instruction; wherein the third behavioral condition includes at least one of the following: the doctor's gesture is either fingers opening and closing or fingers closing; or the oral scanning device is held horizontally and moved vertically after pressing the second preset button; or the head-mounted display is displaced to change the distance between itself and the specified target; the third control instruction is configured to trigger the scaling of the tooth model; the scaling type includes shrinking and enlarging; when the doctor's gesture is fingers opening and closing, or the vertical movement direction of the oral scanning device is upward, or the distance between the head-mounted display and the specified target is shrinking, the tooth model is enlarged; when the doctor's gesture is fingers closing, or the vertical movement direction of the oral scanning device is downward, or the distance between the head-mounted display and the specified target is increasing, the tooth model is shrunk.

[0082] Optionally, the instruction execution unit 200 is specifically configured such that: when the oral scanning device is placed horizontally and moved vertically after pressing the second preset button to control the scaling of the tooth model, the scaling scale is displayed on the side view of the head display, and when the second preset button is released, the scaling scale locks the scaling ratio; after the second preset button is repeatedly pressed and released, the scaling ratio reaches the limit value.

[0083] Optionally, the instruction execution unit 200 is specifically configured to: if the somatosensory interaction information meets the fourth behavioral condition, trigger the oral scanning software in the head-mounted display to execute the fourth control instruction; wherein, the fourth behavioral condition includes at least one of the following: the doctor's hand gesture is a fist with the palm facing up; or the oral scanning device is held in a vertical position after shaking it up and down a preset number of times and then continuously clicking the third preset button; the fourth control instruction is configured to trigger the tooth model to switch to the jaw open state.

[0084] Optionally, the instruction execution unit 200 is specifically configured to: if the somatosensory interaction information meets the fifth behavior condition, trigger the oral scanning software in the head-mounted display to execute the fifth control instruction; wherein, the fifth behavior condition includes at least one of the following: the doctor's hand gesture is an open palm facing upwards; or the oral scanning device is held in a vertical position after shaking it up and down a preset number of times and then continuously clicking the fourth preset button; the fifth control instruction is configured to trigger the tooth model to switch to the jaw closure state.

[0085] Optionally, the instruction execution unit 200 is specifically configured as follows: the augmented reality environment also includes a finger model and an oral scanning device model. The finger model is determined based on the doctor's hand posture captured by the head-mounted display, and the oral scanning device model is determined based on the oral scanning device posture captured by the head-mounted display. When the model editing interface is the tooth position marking function interface, if the head-mounted display's viewing angle meets the requirements, the tooth position label is attached to the position corresponding to the fingertip of the finger model or the top of the oral scanning device model. If the head-mounted display's viewing angle does not meet the requirements, the tooth position label is attached to the position corresponding to the extension line of the finger model or the extension line of the oral scanning device model.

[0086] Optionally, the instruction execution unit 200 is specifically configured to: when the model editing interface is a hole-cutting editing function interface or a brush painting function interface, if the haptic interaction information meets the sixth behavior condition, trigger the oral scanning software in the head-mounted display to execute the sixth control instruction; wherein, the sixth behavior condition is: the doctor's gesture is a circle made of thumb and index finger; the sixth control instruction is configured to trigger the tooth model to revert to the previous operation state.

[0087] Optionally, the instruction execution unit 200 is specifically configured to: when the model editing interface is a hole-cutting editing function interface or a brush painting function interface, if the haptic interaction information meets the seventh behavior condition, trigger the oral scanning software in the head-mounted display to execute the seventh control instruction; wherein, the seventh behavior condition is: the doctor's gesture is a thumb and index finger check; the seventh control instruction is configured to trigger the tooth model to advance to the next operation state.

[0088] Optionally, the instruction execution unit 200 is specifically configured to: when the model editing interface is a custom rotation center interface, generate the rotation center of the tooth model in the augmented reality environment, trigger the selection of the rotation center based on the alignment direction of the finger model or the oral scanning device model, and determine the position of the rotation center based on the drag event of the finger model or the oral scanning device model.

[0089] Each unit described above uses the haptic interaction information collected by the head-mounted display to control the intraoral scanning software within the head-mounted display to execute corresponding control commands, thereby enabling the editing and control of the dental model. This allows doctors to see an intuitive and accurate 3D model in their head and conveniently and quickly edit the dental model using haptic interaction, thus abandoning the traditional mouse, keyboard and screen method and using augmented reality to complete the editing and control of the dental model.

[0090] This application also provides an augmented reality-based model control system, which includes a head-mounted display and an oral scanning device; wherein the head-mounted display is configured to execute the augmented reality-based model control method of this application embodiment; and the oral scanning device is configured to collect patient gingival data and send the patient gingival data to the head-mounted display.

[0091] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0092] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application. Industrial applicability

[0093] The technical solution provided in this application generates a tooth model corresponding to the patient's gingival data collected by an intraoral scanning device, based on the augmented reality environment of a head-mounted display. When the augmented reality environment enters the model editing interface, the intraoral scanning software within the head-mounted display is triggered to execute corresponding control commands based on the haptic interaction information collected by the head-mounted display, causing the tooth model to change in accordance with the control commands. This application utilizes the haptic interaction information collected by the head-mounted display to control the intraoral scanning software within the head-mounted display to execute corresponding control commands, thereby realizing the editing control of the tooth model. This allows doctors to see an intuitive and accurate 3D model in their head, and simultaneously use haptic interaction to conveniently and quickly edit the tooth model, thus abandoning the traditional mouse, keyboard, and screen method and using augmented reality to complete the editing control of the tooth model.

Claims

1. A method for model control based on augmented reality, comprising: generating a tooth model corresponding to patient gingival data collected by an intraoral scanner in an augmented reality environment based on a head-mounted display; when the augmented reality environment enters a model editing interface, triggering an intraoral scanning software in the head-mounted display to execute a corresponding control instruction based on somatosensory interaction information collected by the head-mounted display, so that the tooth model changes in a manner matching the control instruction; wherein the somatosensory interaction information comprises one or more of a doctor gesture, an intraoral scanner holding posture, a head posture, and a head-mounted display displacement trajectory; the triggering condition of the control instruction is that the somatosensory interaction information takes a target gesture or a target intraoral scanner holding posture as a starting behavior; and the intraoral scanning software is a system software supporting editing of the tooth model.

2. The method of claim 1, wherein, in response to the somatosensory interaction information satisfying a first behavior condition, triggering the intraoral scanning software in the head-mounted display to execute a first control instruction; wherein the first behavior condition comprises at least one of the following: the doctor gesture is a hand waving; or the intraoral scanner holding posture is rotating the intraoral scanner; or the head posture is rotating the head; or the head-mounted display displacement trajectory is a circumferential trajectory; and the first control instruction is configured to trigger the tooth model to rotate. in response to the somatosensory interaction information satisfying a second behavior condition, triggering the intraoral scanning software in the head-mounted display to execute a second control instruction; wherein the second behavior condition comprises at least one of the following: the doctor gesture is a palm open with fingers together and palm facing forward; or the intraoral scanner holding posture is vertically holding the intraoral scanner after pressing a first preset button and translating the intraoral scanner; or the head-mounted display displacement trajectory is a translation trajectory; and the second control instruction is configured to trigger the tooth model to translate. in response to the somatosensory interaction information satisfying a third behavior condition, triggering the intraoral scanning software in the head-mounted display to execute a third control instruction; wherein the third behavior condition comprises at least one of the following: the doctor gesture is a finger opening and closing or fingers together; or the intraoral scanner holding posture is horizontally placing the intraoral scanner and vertically moving after pressing a second preset button; or the head-mounted display displaces to change the distance from a specified target; and the third control instruction is configured to trigger the tooth model to zoom; 3. The method of claim 1, wherein, the type of zooming includes zooming out and zooming in; when the doctor gesture is a finger opening and closing, or the vertically moving direction of the intraoral scanner holding posture is upward, or the distance between the head-mounted display and the specified target is reduced, the tooth model is zoomed in; when the doctor gesture is fingers together, or the vertically moving direction of the intraoral scanner holding posture is downward, or the distance between the head-mounted display and the specified target is expanded, the tooth model is zoomed out. when the tooth model is zoomed by horizontally placing the intraoral scanner and vertically moving after pressing the second preset button, a zoom scale is displayed on the side view of the head-mounted display, and when the second preset button is released, the zoom scale locks the zooming magnification; after the second preset button is repeatedly pressed and released, the zooming magnification reaches a limit value. ​ ​ 4. The method of claim 1, wherein, ​ ​ ​ ​ ​ ​ 5. The method of claim 4, wherein, ​ ​ 6. The method of claim 1, wherein, in response to the somatosensory interaction information satisfying a fourth behavior condition, triggering the intra-headset oral scanning software to execute a fourth control instruction; wherein the fourth behavior condition at least includes one of the following: the doctor's gesture is a fist with the palm facing upwards; or the oral scanning device hand-held posture is a vertical continuous clicking on a third preset button after shaking in the up-down direction for a preset number of times; the fourth control instruction is configured to trigger the dental model to switch to a jaw opening state.

7. The method of claim 1, wherein, in response to the somatosensory interaction information satisfying a fifth behavior condition, triggering the intra-headset oral scanning software to execute a fifth control instruction; wherein the fifth behavior condition at least includes one of the following: the doctor's gesture is a hand open with the palm facing upwards; or the oral scanning device hand-held posture is a vertical continuous clicking a fourth preset button after shaking in the up-down direction for a preset number of times; the fifth control instruction is configured to trigger the dental model to switch to a jaw closing state.

8. The method of any one of claims 1-7, wherein, The augmented reality environment further includes a finger model and an oral scanning device model, the finger model being determined based on the doctor's hand posture collected by the headset, and the oral scanning device model being determined based on the oral scanning device posture collected by the headset; When the model editing interface is a tooth position marking function interface, if the view angle of the headset meets the requirements, attach a tooth position label to the position corresponding to the fingertip part of the finger model or the top part of the oral scanning device model, if the view angle of the headset does not meet the requirements, attach the tooth position label to the position corresponding to the extension line of the finger model or the extension line of the oral scanning device model.

9. The method of any one of claims 1-8, wherein, When the model editing interface is a hole digging editing function interface or a brush smearing function interface, in response to the somatosensory interaction information satisfying a sixth behavior condition, triggering the intra-headset oral scanning software to execute a sixth control instruction; wherein the sixth behavior condition is that the doctor's gesture is a thumb and index finger ring; the sixth control instruction is configured to trigger the dental model to revert to the previous operation state.

10. The method of any one of claims 1-9, wherein, When the model editing interface is a hole digging editing function interface or a brush smearing function, in response to the somatosensory interaction information satisfying a seventh behavior condition, triggering the intra-headset oral scanning software to execute a seventh control instruction; wherein the seventh behavior condition is that the doctor's gesture is a thumb and index finger hook; the seventh control instruction is configured to trigger the dental model to advance to the next operation state.

11. The method of claim 8, wherein, When the model editing interface is a self-defined rotation center interface, the augmented reality environment generates a rotation center of the dental model, triggers the rotation center to be selected based on the alignment direction of the finger model or the oral scanning device model, and determines the position of the rotation center based on the dragging event of the finger model or the oral scanning device model.

12. A headset, comprising: a model generation unit configured to generate a dental model corresponding to patient gum data collected by an oral scanning device based on an augmented reality environment of the headset; The instruction execution unit is configured to, when the augmented reality environment enters a model editing interface, trigger the intra-headset oral scanning software to execute corresponding control instructions based on somatosensory interaction information collected by the headset, so that the tooth model changes in a manner matched with the control instructions; wherein the somatosensory interaction information includes one or more of a doctor's gesture, an oral scanning device hand-held posture, a head posture, and a headset displacement trajectory; the trigger condition of the control instructions is that the somatosensory interaction information takes a target gesture or a target oral scanning device hand-held posture as a starting behavior; and the oral scanning software is a system software supporting editing of the tooth model.

13. An augmented reality-based model control system, comprising a headset and an oral scanning device; wherein The headset is configured to perform the augmented reality-based model control method of any one of claims 1-11; and the oral scanning device is configured to collect patient gum data and send the patient gum data to the headset.

Images