Oral treatment unit training system and method based on spatial computing technique

Abstract

The present application relates to an augmented reality technique. Provided are an oral treatment unit training system and method based on a spatial computing technique, which system and method are used for solving the problem of low simulation fidelity due to lack of full-process doctor-patient communication because existing simulators only support single-user operation and provide training based solely on an oral model. By means of the spatial computing technique, a virtual-real fusion head-mounted viewer acquires localization data of an entity and a dental chair of a treatment unit in a real space, and a situational awareness server deploys virtual entities and generates a scene file. A simulation computer generates a virtual-real fusion treatment scene, a virtual patient and treatment tools on the basis of the scene file, and can simulate an entire process from the entry of the patient to the completion of the treatment; and the simulation computer generates haptic information, and controls a haptic tool in the haptic information and a virtual treatment tool in such a way that the poses thereof match that of a haptic feedback tool. A user wears the virtual-real fusion head-mounted viewer to perform a treatment operation. By means of a network synchronization technique, multiple persons viewing an operation process at the same treatment unit is supported. The present application improves the overall simulation fidelity.

Description

A Training System and Method for Dental Treatment Tables Based on Spatial Computing Technology

[0001] This application claims priority to Chinese Patent Application No. 202411573787.0, filed on November 5, 2024, entitled “Training System and Method for Dental Treatment Table Based on Spatial Computing Technology”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of augmented reality technology, and in particular to a dental treatment table training system and method based on spatial computing technology. Background Technology

[0003] With the rapid development of augmented reality technology, it has been widely used in many fields. By establishing a dental treatment table training system using augmented reality technology, doctors can be effectively helped to train in dental treatment.

[0004] In the prior art, there exists an oral surgery skills training simulator, which includes a simulation training platform, an oral operation training system based on a force feedback device, and an observation system based on an augmented reality helmet. The simulation training platform is used to generate a simulated head model, the oral operation training system based on the force feedback device is used to generate specific forces to simulate the force sensation when touching the oral cavity, and the observation system based on the augmented reality helmet is used to generate corresponding virtual training images based on the simulated head model and the force sensation.

[0005] However, existing technologies differ from real clinical procedures, resulting in low simulation realism and impacting user experience. Summary of the Invention

[0006] This application provides a training system and method for dental treatment tables based on spatial computing technology, which aims to improve the realism of simulation.

[0007] In the first aspect, this application proposes a dental treatment table training system based on spatial computing technology, including: multiple treatment tables and a situation observation server, wherein the multiple treatment tables and the situation observation server communicate via TCP / IP protocol, and each treatment table includes: a simulation computer, a virtual-real fusion head-mounted observation device, and a two-hand force feedback tool;

[0008] The virtual-real fusion head-mounted observer is used to acquire the positioning data of physical objects and dental chairs existing in the real space based on spatial computing technology. The physical objects do not include the dental chairs.

[0009] The situation observation server is used to generate a scene file for oral diagnosis and treatment based on the positioning data.

[0010] The simulation computer is used to load the scene file and generate a virtual-real fusion space based on the scene file. In the virtual-real fusion space, a pre-created virtual patient to be diagnosed and a virtual medical tool corresponding to the real medical tool are loaded, and the simulation entity data of the virtual patient and the virtual medical tool are recorded.

[0011] The simulation computer is further configured to generate a virtual diagnosis and treatment scenario based on the scene file and the simulation entity data, and generate force information based on the diagnosis and treatment operation performed by a real user through the operation of the two-hand force feedback tool. The force information includes the pose information of the force feedback tool. Based on a matching algorithm, the force feedback tool and the virtual diagnosis and treatment tool are matched with the pose of the two-hand force feedback tool to obtain the matched simulation entity data.

[0012] The virtual-real fusion head-mounted observer is used to generate a first virtual-real fusion diagnostic image based on the matched simulated entity data. The first virtual-real fusion diagnostic image is used by real users to perform diagnostic operations.

[0013] The situation observation server is also used to acquire simulation entity data corresponding to any treatment station.

[0014] The virtual-real fusion head-mounted observer is also used to generate a second virtual-real fusion medical image based on the simulated entity data of any medical station forwarded by the situation observation server.

[0015] Both the first and second virtual-real fusion diagnostic images show a virtual patient closely positioned on a dental chair in the real space, receiving a diagnostic procedure.

[0016] Optionally, when the virtual-real fusion head-mounted observer is used to acquire positioning data of physical objects and dental chairs existing in real space based on spatial computing technology, it is specifically used for:

[0017] Scan the location images with significant features at a specified location in real space to determine the reference coordinate system and reference points;

[0018] Obtain the pose information of the reference point in the current device coordinate system;

[0019] Based on the pose information of the reference point in the current device coordinate system, establish a mapping relationship between the reference coordinate system and the device coordinate system;

[0020] Measure the positioning data of physical objects existing in real space in the reference coordinate system;

[0021] Scan a positioning image with significant features at the positioning reference point of the dental chair in real space to obtain the pose information of the dental chair in the device coordinate system;

[0022] Based on the pose information of the dental chair in the device coordinate system and the mapping relationship, the positioning data of the dental chair in the reference coordinate system is obtained.

[0023] Optionally, when the situation observation server generates a scene file for oral diagnosis and treatment based on the positioning data, it is specifically used for:

[0024] A virtual space is generated based on the positioning data of the entity object in the reference coordinate system;

[0025] Based on the positioning data of the dental chair in the reference coordinate system, a virtual dental chair is generated at the corresponding position in the virtual space.

[0026] In response to user actions, preset virtual facilities are generated within the virtual space;

[0027] The coordinate and orientation data of the virtual space, the virtual dental chair, and the virtual facilities are obtained, and a scene file for oral diagnosis and treatment is generated based on the coordinate and orientation data.

[0028] Optionally, when the simulation computer generates force information based on a real user's diagnostic operation using the bi-handed force feedback tool, the force information including the pose information of the force feedback tool, and controls the force feedback tool and the virtual diagnostic tool to match the pose of the bi-handed force feedback tool based on a matching algorithm to obtain matched simulation entity data, it is specifically used for:

[0029] Based on the preset position of the dental chair, determine the reference point and force coordinate system of the force coordinate system;

[0030] Obtain the pose information of the head model of the virtual patient in the force coordinate system;

[0031] The force feedback tool performs diagnostic operations based on the head model's pose information in the force feedback coordinate system, generating force feedback information, which includes the pose information of the force feedback tool.

[0032] Acquire the pose information of the virtual diagnostic tool in the force-sensing coordinate system;

[0033] In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the bi-hand force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the matched reference coordinate system.

[0034] Optionally, when the virtual-real fusion head-mounted observer is used to generate a first virtual-real fusion diagnostic image based on the matched simulated entity data, it is specifically used for:

[0035] Obtain the mapping relationship between the reference coordinate system and the device coordinate system;

[0036] Based on the mapping relationship, the simulation entity data in the matched reference coordinate system is transferred to the device coordinate system to obtain the simulation entity data in the device coordinate system.

[0037] The scene file is obtained, and a virtual image in the virtual-real fusion space is obtained based on the scene file and the simulation entity data in the device coordinate system.

[0038] Acquire real-time images of the actual space captured in the photograph;

[0039] The virtual image in the virtual-real fusion space is superimposed with the real image to obtain the first virtual-real fusion diagnostic image.

[0040] Optionally, the situation observation server is also used for:

[0041] Displays virtual treatment scenarios corresponding to different treatment stations.

[0042] Optionally, the situation observation server is also used for:

[0043] The system sets up a virtual patient call mode, which includes a normal call mode and an enhanced call mode. The virtual patients in the normal call mode have random causes of illness, while the virtual patients in the enhanced call mode have preset fixed causes of illness.

[0044] Edit the virtual patient queue corresponding to the aforementioned queuing mode.

[0045] Optionally, the examination table further includes: a speaker and a microphone;

[0046] Once the simulation computer loads a pre-created virtual patient in the virtual-real fusion space, the real user sends control commands to the virtual patient through the microphone and receives interactive voice output by the virtual patient through the speaker.

[0047] Optionally, the treatment table further includes: a height sensor and a pitch sensor;

[0048] When the simulation computer detects a change in the height of the dental chair measured by the height sensor, or a change in the rotation angle of the dental chair measured by the pitch sensor, the simulation computer, based on the matching algorithm, controls the force sensing tool and the virtual diagnostic tool to re-match the poses of the two-hand force feedback tool to obtain new matched simulation entity data.

[0049] Optionally, the oral treatment table training system based on spatial computing technology is connected to at least one external handheld terminal to display virtual-real fusion treatment images on the at least one external handheld terminal, and the at least one external terminal has augmented reality functionality.

[0050] Secondly, this application proposes a dental diagnosis and treatment training method based on spatial computing technology, applied to a dental diagnosis and treatment training system based on spatial computing technology, comprising:

[0051] The positioning data of physical objects and dental chairs existing in real space are obtained based on spatial computing technology, wherein the physical objects do not include the dental chairs.

[0052] A scene file for oral diagnosis and treatment is generated based on the location data;

[0053] Load the scene file and generate a virtual-real fusion space based on the scene file. In the virtual-real fusion space, load the pre-created virtual patient to be treated and the virtual medical tool corresponding to the real medical tool, and record the simulation entity data of the virtual patient and the virtual medical tool.

[0054] A virtual diagnosis and treatment scenario is generated based on the scenario file and the simulation entity data. Force information is generated based on the diagnosis and treatment operation performed by a real user using a two-hand force feedback tool. The force information includes the pose information of the force feedback tool. Based on a matching algorithm, the pose of the force feedback tool and the virtual diagnosis and treatment tool is matched with that of the two-hand force feedback tool to obtain the matched simulation entity data.

[0055] Based on the matched simulation entity data, a first virtual-real fusion diagnostic and treatment image is generated, which is used by real users to perform diagnostic and treatment operations.

[0056] Obtain the simulation entity data corresponding to any treatment station;

[0057] Based on the simulated entity data of any of the forwarded diagnostic and treatment stations, a second virtual-real fusion diagnostic and treatment image is generated;

[0058] Both the first and second virtual-real fusion diagnostic images show a virtual patient closely positioned on a dental chair in the real space, receiving a diagnostic procedure.

[0059] Optionally, the step of acquiring the positioning data of physical objects and dental chairs existing in real space based on spatial computing technology includes:

[0060] Scan the location images with significant features at a specified location in real space to determine the reference coordinate system and reference points;

[0061] Obtain the pose information of the reference point in the current device coordinate system;

[0062] Based on the pose information of the reference point in the current device coordinate system, establish a mapping relationship between the reference coordinate system and the device coordinate system;

[0063] Measure the positioning data of physical objects existing in real space in the reference coordinate system;

[0064] Scan a positioning image with significant features at the positioning reference point of the dental chair in real space to obtain the pose information of the dental chair in the device coordinate system;

[0065] Based on the pose information of the dental chair in the device coordinate system and the mapping relationship, the positioning data of the dental chair in the reference coordinate system is obtained.

[0066] Optionally, generating a scene file for oral diagnosis and treatment based on the positioning data includes:

[0067] A virtual space is generated based on the positioning data of the entity object in the reference coordinate system;

[0068] Based on the positioning data of the dental chair in the reference coordinate system, a virtual dental chair is generated at the corresponding position in the virtual space.

[0069] In response to user actions, preset virtual facilities are generated within the virtual space;

[0070] The coordinate and orientation data of the virtual space, the virtual dental chair, and the virtual facilities are obtained, and a scene file for oral diagnosis and treatment is generated based on the coordinate and orientation data.

[0071] Optionally, the step of generating force information based on real users performing diagnostic operations by operating the bi-handed force feedback tool, the force information including the pose information of the force feedback tool, and controlling the pose of the force feedback tool and the virtual diagnostic tool to match with the bi-handed force feedback tool based on a matching algorithm, to obtain matched simulation entity data, including:

[0072] Based on the preset position of the dental chair, determine the reference point and force coordinate system of the force coordinate system;

[0073] Obtain the pose information of the head model of the virtual patient in the force coordinate system;

[0074] The force feedback tool performs diagnostic operations based on the head model's pose information in the force feedback coordinate system, generating force feedback information, which includes the pose information of the force feedback tool.

[0075] Acquire the pose information of the virtual diagnostic tool in the force-sensing coordinate system;

[0076] In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the bi-hand force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the matched reference coordinate system.

[0077] Optionally, generating a first virtual-real fusion diagnostic image based on the matched simulation entity data includes:

[0078] Obtain the mapping relationship between the reference coordinate system and the device coordinate system;

[0079] Based on the mapping relationship, the simulation entity data in the matched reference coordinate system is transferred to the device coordinate system to obtain the simulation entity data in the device coordinate system.

[0080] The scene file is obtained, and a virtual image in the virtual-real fusion space is obtained based on the scene file and the simulation entity data in the device coordinate system.

[0081] Acquire real-time images of the actual space captured in the photograph;

[0082] The virtual image in the virtual-real fusion space is superimposed with the real image to obtain the first virtual-real fusion diagnostic image.

[0083] This application provides a training system and method for dental treatment tables based on spatial computing technology. The system includes multiple treatment tables and a situation observation server. The multiple treatment tables communicate with the situation observation server via TCP / IP protocol. Each treatment table includes a simulation computer, a virtual-real fusion head-mounted observer, and a two-handed force feedback tool. The virtual-real fusion head-mounted observer uses spatial computing technology to acquire positioning data of physical objects existing in real space and the dental chair / treatment table. The situation observation server generates a scene file for dental treatment based on this positioning data. The simulation computer loads the scene file and generates a virtual-real fusion space based on it. In the virtual-real fusion space, a pre-created virtual patient to be treated and virtual treatment tools corresponding to real treatment tools are loaded, and the simulation entity data of the virtual patient and virtual treatment tools are recorded. The simulation computer also generates a virtual treatment scene based on scene files and simulation entity data. It acquires force information from real users operating a two-handed force feedback tool during treatment, including the tool's pose information. Then, based on a matching algorithm, it controls the force feedback tool and the virtual treatment tool to match the poses of the two-handed force feedback tool, obtaining matched simulation entity data. A virtual-real fusion head-mounted observer generates a first virtual-real fusion treatment image based on this matched simulation entity data. The situation observation server can also acquire simulation entity data corresponding to any treatment station, allowing the virtual-real fusion head-mounted observer to generate a corresponding second virtual-real fusion treatment image based on the simulation entity data of any treatment station forwarded by the situation observation server. This application improves the realism of the simulation by generating a virtual-real fusion treatment scene, and the system supports multiple users watching the treatment process at the same treatment station, facilitating user communication and enhancing the user experience. Attached Figure Description

[0084] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0085] Figure 1 is a schematic diagram of the structure of a dental treatment table training system based on spatial computing technology provided in an embodiment of this application;

[0086] Figure 2 is a flowchart illustrating a method for obtaining positioning data of physical objects and dental chairs in real space based on spatial computing technology, according to an embodiment of this application.

[0087] Figure 3 is a flowchart illustrating a method for obtaining matched simulation entity data according to an embodiment of this application;

[0088] Figure 4 is a schematic diagram of virtual-real matching provided in an embodiment of this application;

[0089] Figure 5 is a schematic diagram of a dental treatment table training system based on spatial computing technology provided in an embodiment of this application;

[0090] Figure 6 is a schematic diagram of the application of a dental treatment table training system based on spatial computing technology provided in an embodiment of this application;

[0091] Figure 7 is a schematic diagram of a scenario provided by an embodiment of this application;

[0092] Figure 8 is a flowchart illustrating an oral diagnosis and treatment training method based on spatial computing technology provided in an embodiment of this application.

[0093] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0094] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0095] In the description of the embodiments of this application, the terms "inner" and "outer", etc., which indicate the direction or positional relationship, are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0096] In the description of the embodiments of this application, unless otherwise expressly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application according to the specific circumstances.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0098] Mastering oral clinical skills requires medical students or doctors to undergo extensive practical training. However, in traditional teaching, medical students have limited opportunities for hands-on practice. Junior doctors often face the dilemma of lacking patients for practical practice in the early stages of their careers, which poses a challenge to their skill development and confidence building.

[0099] In existing technologies, to improve the diagnostic and treatment skills of medical students or doctors, training is typically conducted using surgical skills training simulators to simulate diagnostic and treatment scenarios. These simulators usually include a support platform, a simulation computer, a display screen, and force feedback devices. The support platform has a specific structure that integrates all the devices into a unified simulator. The simulation computer, built into the support platform, provides simulation control of the surgical procedure and outputs visual and force information. The output virtual 3D model of the oral cavity is displayed on the screen for the user to view, while the force information is transmitted through the force feedback device and its series of linkages, ultimately perceived by the user's handheld device (a dental instrument) at the end of the linkages. This handheld device is typically located below the display screen, perceptually in the same space as the virtual 3D model of the oral cavity displayed on the screen, thus simulating the real diagnostic and treatment process.

[0100] For example, there is currently an oral surgery skills training simulator, which includes a simulation training platform, an oral operation training system based on a force feedback device, and an observation system based on an augmented reality helmet. The simulation training platform is used to generate a simulated head model, the oral operation training system based on the force feedback device is used to generate specific forces to simulate the force sensation when touching the oral cavity, and the observation system based on the augmented reality helmet is used to generate corresponding virtual training images based on the simulated head model and the force sensation.

[0101] However, current training simulators only support single-user use and cannot allow multiple people to watch the same treatment operation at the same examination table at the same time. This makes it inconvenient for users to communicate and interact in a timely manner. Furthermore, training is based solely on oral models and does not take into account the patient reception and communication process. There is a lack of practice in the skills of communicating and diagnosing with patients. In addition, the entire virtual space of the examination room is not constructed, resulting in low simulation realism and a lack of immersive experience, which affects the user experience.

[0102] Therefore, addressing the aforementioned technical problems in the prior art, the inventors discovered during their research that, based on spatial computing technology, by overlaying a virtual patient within the real space and fusing the real and virtual spaces, the resulting fused virtual-real diagnostic images can be more realistic, thus improving the simulation's fidelity. Furthermore, allowing multiple users to simultaneously view the fused virtual-real diagnostic images on the same treatment table facilitates timely communication and enhances the user experience. Based on this, this application proposes a dental treatment table training system and method based on spatial computing technology.

[0103] The application scenarios of the oral treatment table training system based on spatial computing technology provided in this application can be used in schools for teaching, in hospitals for training, or in other scenarios where there is a need. This application does not limit the application.

[0104] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0105] Figure 1 is a schematic diagram of the structure of a dental treatment table training system based on spatial computing technology provided in an embodiment of this application. As shown in Figure 1, the system includes: multiple treatment tables 01 and a situation observation server 02. Each treatment table 01 communicates with the situation observation server 02 through the TCP / IP protocol.

[0106] Each treatment station includes: a simulation computer 011, a virtual-real fusion head-mounted observation device 012, and a two-hand force feedback tool 013.

[0107] Virtual-Real Fusion Headset 012: Used to acquire the positioning data of physical objects and dental chair 01 existing in the real space based on spatial computing technology. The physical objects do not include dental chair 01.

[0108] In this embodiment, the virtual-real fusion head-mounted observer 012 can be a mixed reality device with spatial computing capabilities, capable of object positioning, generating virtual images and displaying real images overlaid, and having wireless transmission capabilities, such as a mixed display helmet or augmented reality glasses.

[0109] The virtual-real fusion head-mounted observer 012 is equipped with a scene-setting tool based on spatial computing technology. The scene-setting tool is an application component of the spatial computing technology scene-setting system. It is used to measure the position of physical objects existing in the real space, i.e., the real treatment room, and also to locate the pose of the dental chair 01 in the created virtual space.

[0110] Optionally, the physical objects can be walls, floors, ceilings, doors, etc. It is understood that this embodiment does not limit the types of physical objects, and they can also be other things that exist in the real space.

[0111] After acquiring the location data of the physical objects existing in the real space and the dental chair of the treatment table 01, the virtual-real fusion head-mounted observer 012 communicates them to the situation observation server 02 via the TCP / IP protocol.

[0112] Situation Observation Server 02: Used to generate scene files for oral diagnosis and treatment based on positioning data.

[0113] During the scene setup phase, the situation observation server 02 can receive the location data of entity objects and dental chairs on the treatment table 012 sent by the virtual-real fusion head-mounted observer 012, and generate a virtual space based on the location data. It can also set up virtual facilities related to the treatment room in the virtual space, such as medical cabinets and queuing screens.

[0114] The situation observation server 02 can also generate scene files based on the coordinate and attitude data of the generated virtual space, virtual treatment table, dental chair, and virtual facilities, and save them in a preset storage module.

[0115] In this embodiment, the situation observation server 02 can also set a virtual patient calling mode and edit the virtual patient queue corresponding to the calling mode. The calling mode can include a normal calling mode and an enhanced calling mode. Among them, the virtual patients corresponding to the normal calling mode have random causes of illness, while the virtual patients corresponding to the enhanced calling mode have preset fixed causes of illness.

[0116] In the normal queuing mode, all consultation stations 01 share the same virtual patient queue with random causes. When a consultation station 01 calls a number, it can sequentially remove a patient from the shared queue for treatment. In the enhanced queuing mode, each consultation station 01 corresponds to a dedicated queue with a specific cause sequence for intensive practice.

[0117] The situation observation server 02 is also equipped with a large display screen. When the situation observation server 02 receives the call number information sent by the consultation station 01, it can determine the target virtual patient from the virtual patient queue corresponding to the call number mode set by the situation observation server 02 and the identification information of the consultation station 01 included in the call number information, and send it to the corresponding consultation station 01. The large display screen will also display the target virtual patient.

[0118] Simulation Computer 011: Used to load scene files and generate a virtual-real fusion space based on the scene files. In the virtual-real fusion space, pre-created virtual patients to be treated and virtual medical tools corresponding to real medical tools are loaded, and the simulation entity data of virtual patients and virtual medical tools are recorded.

[0119] In this embodiment, the simulation computer 011 can be connected to the situation observation server 02 via a wired connection to communicate and load the scene files distributed by the situation observation server 02.

[0120] An oral diagnosis and treatment simulation training system runs in simulation computer 011.

[0121] The simulation computer 011 loads the scene file distributed by the situation observation server 02, and generates a virtual-real fusion space based on the scene file to simulate a treatment room for diagnosis and treatment training.

[0122] The simulation computer 011 loads a pre-created virtual patient awaiting treatment by performing a call-number operation, and loads the corresponding virtual oral model of the virtual patient. The virtual oral model has been pre-built based on oral data, such as cone-beam CT (CBCT) scan data and true color scan data.

[0123] Optionally, the oral cavity data of the virtual patient can be obtained in advance by collecting oral cavity data from real patients. For example, the patient's oral cavity can be scanned using a CBCT scanner and a true-color scanner to obtain CBCT oral cavity data in DICOM format and oral cavity scan data in STL format. Based on the above data, a complete three-dimensional surface and physical model of the lower skull are obtained by overlaying and reconstructing a pre-defined three-dimensional model, thereby pre-constructing a virtual oral cavity model.

[0124] Optionally, the virtual oral operating environment can be pre-created based on a virtual oral model and virtual diagnostic tools. The virtual diagnostic tools can be pre-created based on the obtained measurement data such as the size and shape of the diagnostic tools.

[0125] Optionally, the virtual patient can be pre-created. Specifically, based on a pre-built instruction-based behavior-driven model and knowledge graph model, a set of diagnostic information for the virtual patient is established, and the set of diagnostic information is integrated with the virtual oral operating environment to ultimately generate the virtual patient, which has force feedback interaction capabilities.

[0126] The simulation computer 011 records the simulation entity data of the virtual patient and the virtual medical tool, and shares the simulation entity data with the situation observation server 02.

[0127] In this embodiment, each simulation computer 011 will be assigned a unique ID. Sim When the simulation computer 011 loads the virtual patient and virtual treatment tools, it will assign a unique local ID to the loaded virtual patient and virtual treatment tools. Local And based on the preset ID Sim <<16|Id Local The method generates globally unique IDs for loaded virtual patients and virtual medical tools. Global It serves as a synchronous identifier for a unified virtual-real integrated space.

[0128] Simulation computer 011: Used to generate virtual diagnosis and treatment scenarios based on scene files and simulation entity data, and to obtain force information based on the real user's operation of the two-hand force feedback tool 013 to perform diagnosis and treatment operations. The force information includes the pose information of the force feedback tool, as well as the magnitude and direction of the force.

[0129] The simulation computer 011 generates a virtual diagnosis and treatment scene based on the loaded scene file and the created simulation entities, where each simulation entity has a different ID. Global Logo.

[0130] The simulation computer 011 is also used to match the poses of force feedback tools and virtual diagnostic tools with bi-hand force feedback tools based on matching algorithms, and to obtain the matched simulation entity data.

[0131] After the virtual patient and virtual diagnostic tools are loaded into the simulation computer 011, force feedback tools 013 are used to perform diagnostic operations on the virtual patient to obtain force information. Then, based on a preset triple matching algorithm, the force feedback tools and the virtual diagnostic tools are matched with the poses of the hand-held force feedback tools, so that the presented virtual and real diagnostic images are closer to reality.

[0132] Then, the virtual-real fusion head-mounted observer 012 generates the first virtual-real fusion diagnostic image based on the matched simulated entity data.

[0133] The virtual-real fusion head-mounted observer 012 first detects whether there is an identifier Id. Global If the corresponding object exists, the existing object is updated based on the obtained simulation entity data; if it does not exist, the corresponding object is created based on the obtained simulation entity data, and finally, the virtual-real fusion diagnostic and treatment image is obtained.

[0134] In this application, the situation observation server 02 can also obtain simulation entity data from any treatment station simulation computer, so that the virtual-real fusion head-mounted observer 012 can generate corresponding virtual-real fusion treatment images based on the simulation entity data of any treatment station forwarded by it.

[0135] The generated virtual-real fusion diagnostic images will show a virtual patient closely attached to the dental chair and examination table in the real space, receiving diagnostic and treatment procedures.

[0136] By using TCP / IP-based communication, i.e., multi-user network synchronization technology, multiple treatment stations can be networked together for use. This allows any real user wearing the virtual-real fusion head-mounted observer 012 to observe the virtual-real fusion medical images of other treatment stations at any time, facilitating timely communication among users and thus improving the user experience.

[0137] In this application, the large display screen equipped with the situation observation server 02 can also display the virtual diagnosis and treatment scene corresponding to any diagnosis and treatment station.

[0138] In the above embodiments of this application, the system includes: multiple treatment tables and a situation observation server. The multiple treatment tables communicate with the situation observation server via TCP / IP protocol. Each treatment table includes: a simulation computer, a virtual-real fusion head-mounted observer, and a two-handed force feedback tool. The virtual-real fusion head-mounted observer uses spatial computing technology to acquire positioning data of physical objects existing in real space and the dental chairs and treatment tables. The situation observation server generates a scene file for oral treatment based on this positioning data. The simulation computer loads the scene file and generates a virtual-real fusion space based on it. In the virtual-real fusion space, a pre-created virtual patient to be treated and virtual treatment tools corresponding to real treatment tools are loaded, and the simulation entity data of the virtual patient and virtual treatment tools are recorded. The simulation computer also generates a virtual treatment scene based on scene files and simulation entity data. It acquires force information from real users operating a two-handed force feedback tool during treatment, including the tool's pose information. Then, based on a matching algorithm, it controls the force feedback tool and the virtual treatment tool to match the poses of the two-handed force feedback tool, obtaining matched simulation entity data. A virtual-real fusion head-mounted observer generates a first virtual-real fusion treatment image based on this matched simulation entity data. The situation observation server can also acquire simulation entity data corresponding to any treatment station, allowing the virtual-real fusion head-mounted observer to generate a corresponding second virtual-real fusion treatment image based on the simulation entity data of any treatment station forwarded by the situation observation server. This embodiment's system improves simulation realism by generating a virtual-real fusion treatment scene. Furthermore, the system supports multiple users viewing the same treatment process at the same station, facilitating communication and enhancing the user experience.

[0139] Furthermore, based on the above embodiments, the following embodiments illustrate the process by which a virtual-real fusion head-mounted observer acquires positioning data of physical objects and dental chairs existing in real space using spatial computing technology.

[0140] Figure 2 is a flowchart illustrating a method for obtaining positioning data of physical objects and dental chairs in real space based on spatial computing technology, according to an embodiment of this application. As shown in Figure 2, the method may include:

[0141] S201. Scan the location image with significant features at the specified location in real space, and determine the reference coordinate system and reference point.

[0142] S202. Obtain the pose information of the reference point in the current device coordinate system.

[0143] In a real empty room R beforehand real Select a location as reference point O. ref It also specifies the orientation of each coordinate axis to form a reference coordinate system, Coord. ref At reference point O ref Place a marker image, which can be a QR code or other image with distinctive features.

[0144] Once the virtual-real fusion head-mounted observer is activated, the device coordinate system (Coord) of the current device is determined. dev and origin O dev .

[0145] The virtual-real fusion head-mounted observer scans the marker image to obtain the reference point O in the reference coordinate system. ref Pose information O in the current device coordinate system ref = (x,y,z,α,β,γ), where x, y, and z are reference points O. ref Location information, α, β, γ are reference points O ref 'The orientation information.

[0146] S203. Based on the pose information of the reference point in the current device coordinate system, establish the mapping relationship between the reference coordinate system and the device coordinate system.

[0147] Combining O ref The pose information of ' is used to construct O using coordinate basis transformation. dev To O ref The translation matrix T and the rotation matrix R are given. Multiplying the two matrices, T·R, yields the reference coordinate system Coord. ref To the device coordinate system Coord dev The transformation matrix M.

[0148] Inverting the transformation matrix M yields Mi -1 The matrix M -1 That is, the device coordinate system Coord dev To the reference coordinate system Coord ref The transformation matrix.

[0149] Transformation matrix M and M -1 This represents the mapping relationship between the reference coordinate system and the device coordinate system.

[0150] S204. Measure the positioning data of physical objects existing in real space in the reference coordinate system.

[0151] Measure physical objects existing in a real space, such as walls, floors, ceilings, doors, etc., relative to a reference point O. ref The location data is used by the situation observation server to construct the virtual space R. virtual This makes the virtual space R virtual The walls, floors, ceilings, doors, etc. in the image are different from the actual space R real The positions of all walls, floors, ceilings, and doors are consistent.

[0152] S205. Scan the positioning image of the dental chair in real space with significant features at the positioning reference point to obtain the position and pose information of the dental chair in the equipment coordinate system.

[0153] In a real empty room R beforehand real The interior is furnished with dental chairs and treatment tables (C1, C2, ... C1) for simulation training. n ), at each examination station C i Different marker diagrams M are placed at the origin. i , of which M i Includes images of real-world dental examination tables and chairs.

[0154] Virtual-Real Fusion Headset Scanning M i Obtain the Marker identifier and the treatment table Ci in the device coordinate system Coord. dev The pose information below.

[0155] S206. Based on the pose information and mapping relationship of the dental chair in the equipment coordinate system, obtain the positioning data of the dental chair in the reference coordinate system.

[0156] The virtual-real fusion head-mounted observer uses the transformation matrix M -1 Transform the pose information of the dental chair in the device coordinate system to the reference coordinate system Coord. ref The data is then combined with the Marker identifier and saved to obtain the positioning data of the dental chair in the reference coordinate system.

[0157] The virtual-real fusion head-mounted observer acquires the positioning data of physical objects and dental chairs in the real space and communicates it to a situational observation server via TCP / IP protocol. The situational observation server generates a virtual space based on the positioning data of the physical objects in the reference coordinate system, and generates a virtual dental chair at the corresponding location in the virtual space based on the positioning data of the dental chair in the reference coordinate system. In response to user actions, it generates preset virtual facilities within the virtual space, thereby acquiring the coordinate and attitude data of the virtual space, virtual dental chair, and virtual facilities, and generating a scene file for oral treatment based on the coordinate and attitude data.

[0158] In the above embodiments of this application, a virtual-real fusion head-mounted observer scans positioning images with significant features at specified locations in real space to determine a reference coordinate system and a reference point, and obtains the pose information of the reference point in the current device coordinate system. Then, based on the pose information of the reference point in the current device coordinate system, a mapping relationship between the reference coordinate system and the device coordinate system is established. The virtual-real fusion head-mounted observer measures the positioning data of physical objects existing in real space in the reference coordinate system, and scans positioning images with significant features at the positioning reference point of the dental chair in real space to obtain the pose information of the dental chair in the device coordinate system. Then, based on the pose information of the dental chair in the device coordinate system and the mapping relationship, the positioning data of the dental chair in the reference coordinate system is obtained. This embodiment establishes a reference coordinate system Coord... ref With the device coordinate system Coord dev The mapping relationship between them facilitates the fusion of virtual and real spatial coordinates when the virtual-real head-mounted observation device performs visual display.

[0159] Furthermore, based on the above embodiments, the following embodiments illustrate the process by which the simulation computer obtains the matched simulation entity data.

[0160] Figure 3 is a flowchart illustrating a method for obtaining matched simulation entity data according to an embodiment of this application. As shown in Figure 3, the method may include the following steps:

[0161] S301. Determine the reference point and force coordinate system of the force coordinate system according to the preset position of the dental chair.

[0162] In a real empty room R beforehand real Select a location as reference point O. ref It also specifies the orientation of each coordinate axis to form a reference coordinate system, Coord. ref ,

[0163] According to the method in the embodiment shown in Figure 2, the mapping relationship between the reference coordinate system and the device coordinate system is established, that is, the transformation matrix M is constructed.

[0164] For any dental chair C in real space i Obtain its world transformation matrix M Ci The matrix contains C i In the reference coordinate system Coord ref The pose information below.

[0165] Select fixed to dental chair C i The center point of the connecting line between the two force feedback ink cartridges on the backrest and the panel is used as the reference point for the force-haptic workspace, denoted as O. Fi O Fi The pose of the dental chair in the local coordinate system is T. L-Ofi In the reference coordinate system Coord ref The next pose is T W-Ofi =M Ci ·T L-Ofi denoted as M W-Ofi . Take O Fi Establish a force perception coordinate system Coord with the origin. Ofi , for M W-Ofi Inverse the equation to get the Coord. ref To Coord Ofi Transformation matrix M Ofi -1 .

[0166] S302 acquires the pose information of the virtual patient's head model in the force coordinate system.

[0167] When constructing a virtual patient with force feedback interaction capabilities, the computer simulation records the local pose information T of the head model's coordinate origin relative to the animated head bone. L-HB .

[0168] The real user sends control commands via microphone, and the simulation computer controls the virtual patient to lie down in the dental chair according to the control commands. i At this time, the simulation computer obtains the Head Bone in the reference coordinate system Coord. ref pose information T W-HB The force sensory head model was calculated in Coord ref The pose information T below H-Ref =T W-HB ·T L-HB Transform it to the Coord coordinate system. Ofi The pose information T in the force coordinate system is obtained. H-Ofi =M Ofi-1 ·T H-Ref .

[0169] S303. Perform diagnostic operations based on the head model's pose information in the force feedback coordinate system using the two-hand force feedback tool to obtain force information.

[0170] The simulation computer will use the pose information of the force-sensing head model to... H-Ofi The force feedback is transmitted to the force feedback module. The real user uses the bi-handed force feedback tool to perform diagnostic operations, which drives the force feedback module to perform force calculations and obtain force information. The force information includes the position and pose information of the force feedback tool, as well as the magnitude and direction of the force. The calculated force information is then fed back to the real user through the bi-handed force feedback tool.

[0171] S304. Obtain the pose information of the virtual diagnostic tool in the force coordinate system.

[0172] Move the handles of the two-handed force feedback tools (the tools for the left and right force feedback devices) from the ink cartridges to O respectively. Fi The pose information of each handle in its device coordinate system is obtained and denoted as T. L T R , for T L T R Inverse each transformation matrix T is obtained by taking their inverses. L -1 T R -1 That is, from the coordinate system of each force feedback device to the force perception coordinate system Coord. Ofi The transformation matrix under the given conditions.

[0173] S305. In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the two-hand force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the reference coordinate system after matching. The matched simulated entity data is used by the virtual-real fusion head-mounted observer to generate virtual-real fusion diagnostic images with better matching effect.

[0174] Using the transformation matrix T L -1 T R -1 The bi-handed force feedback diagnostic tools were transformed into the force perception coordinate system Coord. Ofi The pose information P in the force-sensing coordinate system is obtained. Ti-Ofi , where i can be L or R.

[0175] The pose information P Ti-Ofi Transform to reference coordinate system Coord ref Below, the pose information P in the reference coordinate system is obtained. Ti-Ref =TW-Ofi ·P Ti-Ofi .

[0176] Using the transformation matrix M to transform P Ti-Ref The data is transformed to the coordinate system of the virtual-real fusion head-mounted observer to obtain the simulated entity data in the device coordinate system. The virtual-real fusion head-mounted observer acquires the scene file, and based on the scene file and the simulated entity data, obtains the virtual image in the virtual-real fusion space, as well as the real-time image captured in the real space. The virtual image in the virtual-real fusion space and the real-time image are superimposed to obtain the final virtual-real fusion diagnostic image, thereby realizing the force sensation of the diagnostic tool and the triple matching of the virtual diagnostic tool and the hand force feedback tool in the fusion space.

[0177] As shown in Figure 4, which is a schematic diagram of virtual-real matching provided in an embodiment of this application, in Figure 4, 401 is the virtual patient, 402 is the physical dental chair, 403 is the handle of the right-hand force feedback device, 404 is the right-hand virtual treatment tool, 405 is the handle of the left-hand force feedback device, and 406 is the left-hand virtual treatment tool. As can be seen from Figure 4, the physical force feedback handle and the virtual treatment tool are overlapped and matched in the virtual-real fusion space. In terms of actual tactile force, the virtual treatment tool and the force feedback tool model are also matched. The real user perceives force based on this force feedback tool model through the force feedback devices of both hands.

[0178] In the above embodiments of this application, by determining the reference point and force coordinate system of the force-sensing coordinate system according to the preset position of the dental chair, the pose information of the virtual patient's head model in the force-sensing coordinate system is obtained. Then, based on the pose information of the head model in the force-sensing coordinate system, a bi-handed force feedback tool is used to perform diagnostic operations and obtain force information. The pose information of the virtual diagnostic tool in the force-sensing coordinate system is also obtained. In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the bi-handed force feedback tool, and then transformed to a reference coordinate system to obtain the simulated entity data in the matched reference coordinate system. This embodiment achieves the matching of force space, virtual space, and real space by matching the force-sensing tool and the virtual diagnostic tool with the bi-handed force feedback tool, making the virtual-real fusion diagnostic images generated in the virtual-real fusion head-mounted observer more accurate.

[0179] The following embodiments illustrate a more detailed structural composition of the dental treatment table training system based on spatial computing technology of this application.

[0180] In this embodiment, in addition to the simulation computer 011, the virtual-real fusion head-mounted observer 012, and the two-hand force feedback tool 013, the treatment table 01 also includes: a touch display 014, a dental chair height sensor 015, a pitch sensor 016, a dental chair height / pitch controller panel 017, a speaker 018, a microphone 019, and a retractable headrest assembly 0110.

[0181] As shown in Figure 5, Figure 5 is a schematic diagram of a dental treatment table 01 training system based on spatial computing technology provided in an embodiment of this application:

[0182] The dental chair of the treatment table 01 includes height adjustment and backrest tilt functions. The side is designed as a shell for housing the main unit of the treatment table 01. The mechanical structure of the dental chair can stably support the entire treatment table 01. Two connecting drive rods are used to control the height adjustment and backrest tilt of the dental chair respectively. Each drive rod is connected by a motor and controlled by the height adjustment / tilt controller panel 017.

[0183] The two-handed force feedback tool 013 consists of two force feedback devices, which are fixed to the back of the dental chair and arranged symmetrically along the longitudinal central axis of the dental chair, leaving enough space to prevent physical interference between the devices. They can rise, fall, and tilt synchronously with the dental chair.

[0184] The 014 touch monitor connects to the main unit casing using a 6-axis connector, allowing users to use the monitor at any height, left and right angle, and tilt angle.

[0185] The dental chair height sensor 015 and pitch sensor 016 use tilt sensors to measure the real-time angle between the dental chair support rod and the horizontal plane. The height sensor 015 is fixed on the support rod of the main frame, and the pitch sensor 016 is fixed on the backrest.

[0186] The dental chair height / tilt controller panel is connected to the dental chair main unit via a 2-axis connector, allowing for overall left and right deflection of the controller panel 017 and the connecting rod, as well as rotation of the panel itself, making it convenient for users.

[0187] The speaker 018 and microphone 019 are made from mature industrial speaker and microphone components, and the corresponding mounting positions are designed. The speaker 018 is installed inside the back of the dental chair, and the microphone 019 is installed in the middle of the hand force feedback and has a hole in the panel so that it can receive sound.

[0188] The retractable headrest component 0110 can be adjusted to suit virtual patients of different heights, ensuring that the virtual patient's head is positioned appropriately on the headrest, providing support for real users during operation.

[0189] To facilitate understanding of the dental treatment table training system based on spatial computing technology described above in this application, the application process is illustrated below with specific examples. Figure 6 is a schematic diagram of the application of a dental treatment table training system based on spatial computing technology provided in an embodiment of this application, as shown in Figure 6: including a preparation stage and a training stage.

[0190] During the preparation phase (shown by the dashed line in Figure 6):

[0191] Constructing a virtual-real integrated space, i.e., a virtual-real integrated treatment room:

[0192] The constructed virtual-real integrated treatment room includes virtual objects corresponding to physical objects existing in the real space, virtual treatment tables and dental chairs, as well as virtual medical cabinets, virtual queuing screens, and other virtual objects.

[0193] Building a virtual dental operating environment:

[0194] A virtual oral model is established based on oral data such as CBCT scans and true color scans. A virtual diagnostic and treatment tool is established based on the measurement data of the diagnostic and treatment tool. A virtual oral operating environment is then established based on the virtual oral model and the virtual diagnostic and treatment tool.

[0195] Building virtual patients:

[0196] By utilizing a digital holographic real-time 3D model with voice recognition interaction, a command-based behavior-driven model is defined. This model, combined with a knowledge graph model, establishes a set of diagnostic information for a virtual patient. This information is then fused with a virtual dental operating environment to generate a virtual patient. This virtual patient can receive control commands from a simulation computer to perform actions such as walking, standing, getting in and out of the dental chair, and facial expression control.

[0197] Establish a communication network:

[0198] Multiple diagnostic stations (i.e., multiple simulation computers), virtual-real fusion head-mounted observers, and a situation observation server are connected to each other. The simulation computers can be connected to the situation observation server via wired connections, while the virtual-real fusion head-mounted observers can be connected wirelessly, each establishing a TCP / IP communication link.

[0199] During the simulation training phase (shown by the solid line in Figure 6):

[0200] Real users conducted simulated training using hand-held force feedback tools:

[0201] The simulated computer loads scene files distributed by the situation observation server, generating a virtual-real fusion space. Real users use the touch screen's user interface to call numbers, generating virtual patients. They then use microphones to ask questions of the virtual patients and receive the interactive voice output from the virtual patients through speakers.

[0202] Real users perform diagnostic procedures in a virtual-real fusion space using a two-handed force feedback tool as an operational intermediary. A simulated computer collects the positional data of the two-handed force feedback tool in real time, calculates force information, and then feeds this calculated force information back to the operator (real user) via the tool, allowing the user to experience force within the virtual environment. This force information includes the tool's pose, the magnitude of the force, and its direction.

[0203] The computer simulation performs a triple matching of the poses of force feedback tools, virtual diagnostic tools, and bimanual force feedback tools:

[0204] The simulation computer uses a preset matching algorithm to match the poses of force-sensing tools, virtual diagnostic tools, and bi-handed force feedback tools, ensuring that the poses of the force-sensing and visual virtual diagnostic tools always overlap with those of the bi-handed force feedback tools.

[0205] After adjusting the height and tilt of the dental chair, the virtual patient moves in sync with the chair. When the simulation computer detects a change in the chair's height (measured by the height sensor) or rotation angle (measured by the tilt sensor), the force sensor and virtual treatment tool are re-matched with the real treatment tool's pose based on the aforementioned matching algorithm. At this point, the position of the sensor is re-acquired. Fi New pose T in the local coordinate system of the dental chair L-Ofi ', and update M W-Ofi and M Ofi -1 And update the pose information of the virtual diagnosis and treatment tool in the virtual-real fusion head-mounted observer, and achieve triple matching again to obtain new simulated entity data.

[0206] Simulated entity data network synchronization:

[0207] After obtaining the simulated entity data matched by the simulated computer, the simulation server can process or forward the simulated entity data according to appropriate strategies. Each simulated entity data item will contain a globally unique identifier (Id). Global And other related information, such as entity type, entity pose, simulation mesh deformation data, and animation data.

[0208] Real users can observe the diagnosis and treatment process using a virtual-real fusion head-mounted observation device:

[0209] The virtual-real fusion head-mounted observer scans marker maps to determine the reference coordinate system and reference points, and establishes a mapping relationship between the reference coordinate system and the device coordinate system, i.e., a mapping relationship between virtual space and real space. The scene file is loaded, and the matched simulation entity data sent by the simulation computer is received to obtain a virtual image in the virtual-real fusion space. Finally, the real-time image of the real space is superimposed to obtain a virtual-real fusion diagnostic and therapeutic image.

[0210] Real users can observe the diagnosis and treatment process using a handheld tablet:

[0211] In this application, the dental treatment table system based on spatial computing technology can also be connected to at least one external handheld terminal to display a fusion of virtual and real diagnostic images. The external handheld terminal has augmented reality capabilities and is wirelessly connected to a simulation computer. Taking a handheld tablet as an example, the external handheld terminal scans a marker map to determine a reference coordinate system and reference points, and establishes a mapping relationship between the reference coordinate system and the device coordinate system, i.e., a mapping relationship between virtual space and real space. It loads a scene file and receives matched simulation entity data sent by the simulation computer to obtain a virtual image in the fusion space. Finally, the real-time image of the real space is superimposed to obtain the fusion of virtual and real diagnostic images.

[0212] This application does not limit the number of external handheld terminals that can be connected; there can be one or more.

[0213] The treatment process has ended.

[0214] After the diagnosis and treatment process is completed, the simulation computer can also control the virtual patient to leave the examination table and provide a systematic evaluation of the real user's diagnosis and treatment operation in the virtual-real integrated space.

[0215] In this application, multiple treatment stations are connected to a situation monitoring server, as shown in Figure 7. Figure 7 is a schematic diagram of a scenario provided by an embodiment of this application. In Figure 7:

[0216] Assume there are four treatment stations in the actual treatment space, connected to a situation observation server 02 via a local area wired network 03. The situation observation server 02 is equipped with a large display screen. Station 101 is treatment station 1, station 102 is the virtual patient at treatment station 1, and station 103 is the real user at treatment station 1. Similarly, station 201 is treatment station 2, station 202 is the virtual patient at treatment station 2, and station 203 is the real user at treatment station 2. Station 301 is treatment station 3, station 302 is the virtual patient at treatment station 3, station 303 is the real user at treatment station 3, and station 304 is another real user at this station. In treatment station 4, in addition to stations 401, 402, and 403 (the virtual patient and real user), station 404 is also included. Station 404 is the teacher, who can observe the treatment process at treatment station 4 via a handheld tablet computer 405.

[0217] In this application, through the situation observation server, real users can observe the treatment process of any treatment station and control the "camera" function on the large display screen of the situation observation server to move the "camera" to the preset position of the corresponding treatment station for viewing.

[0218] The dental treatment table training system based on spatial computing technology proposed in this application improves the realism of simulation by generating a virtual-real fusion treatment scene. In addition, the system can support multiple people to watch the treatment process on the same treatment table, which facilitates communication among users and enhances the user experience.

[0219] This application also provides a dental diagnosis and treatment training method based on spatial computing technology, applied to a dental diagnosis and treatment training system based on spatial computing technology, as shown in Figure 8. Figure 8 is a flowchart illustrating a dental diagnosis and treatment training method based on spatial computing technology provided in an embodiment of this application. The method includes:

[0220] S801. Based on spatial computing technology, obtain the positioning data of physical objects and dental chairs existing in the real space. The physical objects do not include dental chairs.

[0221] One possible implementation is:

[0222] The process involves scanning location images with significant features at specified positions in real space to determine a reference coordinate system and reference points, and obtaining the pose information of the reference points in the current device coordinate system. Based on the pose information of the reference points in the current device coordinate system, a mapping relationship is established between the reference coordinate system and the device coordinate system. Additionally, the process involves measuring the positioning data of physical objects in real space in the reference coordinate system, scanning location images with significant features at the positioning reference point of the dental chair / treatment table in real space to obtain the pose information of the dental chair / treatment table in the device coordinate system, and then obtaining the positioning data of the dental chair / treatment table in the reference coordinate system based on the pose information of the dental chair / treatment table in the device coordinate system and the mapping relationship.

[0223] S802. Generate a scene file for oral diagnosis and treatment based on the positioning data.

[0224] One possible implementation is:

[0225] Based on the positioning data of the physical objects in the reference coordinate system, a virtual space is generated. Based on the positioning data of the dental chair and treatment table in the reference coordinate system, a virtual dental chair and treatment table are generated at the corresponding positions in the virtual space. In response to user actions, preset virtual facilities are generated within the virtual space. The coordinate and orientation data of the virtual space, virtual dental chair and treatment table, and virtual facilities are acquired, and a scene file for oral treatment is generated based on this data.

[0226] S803. Load the scene file and generate a virtual-real fusion space based on the scene file. In the virtual-real fusion space, load the pre-created virtual patient to be treated and the virtual medical tool corresponding to the real medical tool, and record the simulation entity data of the virtual patient and the virtual medical tool.

[0227] S804. Generate a virtual diagnosis and treatment scene based on the scene file and simulation entity data, and generate force information based on the diagnosis and treatment operation performed by the real user through the operation of the two-hand force feedback tool. The force information includes the pose information of the force feedback tool. Based on the matching algorithm, control the force feedback tool and the virtual diagnosis and treatment tool to match the pose of the two-hand force feedback tool to obtain the matched simulation entity data.

[0228] One possible implementation is:

[0229] Based on the preset position of the dental chair, the reference point and force coordinate system are determined, and the pose information of the virtual patient's head model in the force coordinate system is obtained. The bi-handed force feedback tool performs diagnostic operations based on the pose information of the head model in the force coordinate system, generating force information, which includes the pose information of the force feedback tool. The pose information of the virtual diagnostic tool in the force coordinate system is obtained. In the force coordinate system, the pose information of the force feedback tool and the virtual diagnostic tool is matched with the pose information of the bi-handed force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the matched reference coordinate system.

[0230] S805. Based on the matched simulation entity data, generate a first virtual-real fusion diagnostic image, which is used by real users to perform diagnostic operations.

[0231] One possible implementation is:

[0232] The mapping relationship between the reference coordinate system and the device coordinate system is obtained. Based on this mapping relationship, the simulated entity data in the matched reference coordinate system is transferred to the device coordinate system to obtain the simulated entity data in the device coordinate system. A scene file is acquired. Based on the scene file and the simulated entity data in the device coordinate system, a virtual image in the virtual-real fusion space is obtained, along with real-time images captured in the actual space. The virtual image in the virtual-real fusion space and the real-time image are overlaid to obtain the first virtual-real fusion diagnostic image.

[0233] S806. Obtain the simulation entity data corresponding to any treatment station.

[0234] S807. Generate a second virtual-real fusion diagnostic image based on the simulation entity data of any forwarded diagnostic station.

[0235] Both the first and second virtual-real fusion diagnostic images show a virtual patient closely positioned on a dental chair in the real space, receiving a diagnostic procedure.

[0236] The specific implementation process and technical effects of the methods described in this application are detailed in the above embodiments. To avoid redundancy, they will not be described again.

[0237] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0238] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0241] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0243] Finally, it should be noted that other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and alterations may be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training system for an oral treatment table based on spatial computing technology, characterized in that, include: The system includes multiple treatment stations and a situation observation server. The multiple treatment stations communicate with the situation observation server via TCP / IP protocol. Each treatment station includes: a simulation computer, a virtual-real fusion head-mounted observation device, and a two-hand force feedback tool. The virtual-real fusion head-mounted observer is used to acquire the positioning data of physical objects and dental chairs existing in the real space based on spatial computing technology. The physical objects do not include the dental chairs. The situation observation server is used to generate a scene file for oral diagnosis and treatment based on the positioning data. The simulation computer is used to load the scene file and generate a virtual-real fusion space based on the scene file. In the virtual-real fusion space, a pre-created virtual patient to be diagnosed and a virtual medical tool corresponding to the real medical tool are loaded, and the simulation entity data of the virtual patient and the virtual medical tool are recorded. The simulation computer is further configured to generate a virtual diagnosis and treatment scenario based on the scene file and the simulation entity data, and generate force information based on the diagnosis and treatment operation performed by a real user through the operation of the two-hand force feedback tool. The force information includes the pose information of the force feedback tool. Based on a matching algorithm, the force feedback tool and the virtual diagnosis and treatment tool are matched with the pose of the two-hand force feedback tool to obtain the matched simulation entity data. The virtual-real fusion head-mounted observer is used to generate a first virtual-real fusion diagnostic image based on the matched simulated entity data. The first virtual-real fusion diagnostic image is used by real users to perform diagnostic operations. The situation observation server is also used to acquire simulation entity data corresponding to any treatment station. The virtual-real fusion head-mounted observer is also used to generate a second virtual-real fusion medical image based on the simulated entity data of any medical station forwarded by the situation observation server. Both the first and second virtual-real fusion diagnostic images show a virtual patient closely positioned on a dental chair in the real space, receiving a diagnostic procedure.

2. The system according to claim 1, characterized in that, When the virtual-real fusion head-mounted observer is used to acquire positioning data of physical objects and dental chairs existing in real space based on spatial computing technology, it is specifically used for: Scan the location images with significant features at a specified location in real space to determine the reference coordinate system and reference points; Obtain the pose information of the reference point in the current device coordinate system; Based on the pose information of the reference point in the current device coordinate system, establish a mapping relationship between the reference coordinate system and the device coordinate system; Measure the positioning data of physical objects existing in real space in the reference coordinate system; Scan a positioning image with significant features at the positioning reference point of the dental chair in real space to obtain the pose information of the dental chair in the device coordinate system; Based on the pose information of the dental chair in the device coordinate system and the mapping relationship, the positioning data of the dental chair in the reference coordinate system is obtained.

3. The system according to claim 2, characterized in that, When the situation observation server generates a scene file for oral diagnosis and treatment based on the positioning data, it is specifically used for: A virtual space is generated based on the positioning data of the entity object in the reference coordinate system; Based on the positioning data of the dental chair in the reference coordinate system, a virtual dental chair is generated at the corresponding position in the virtual space. In response to user actions, preset virtual facilities are generated within the virtual space; The coordinate and orientation data of the virtual space, the virtual dental chair, and the virtual facilities are obtained, and a scene file for oral diagnosis and treatment is generated based on the coordinate and orientation data.

4. The system according to claim 3, characterized in that, The simulation computer, when used to generate force information based on real users' diagnostic operations using the bi-handed force feedback tool, including the pose information of the force feedback tool, and controlling the force feedback tool and the virtual diagnostic tool to match the pose of the bi-handed force feedback tool based on a matching algorithm to obtain matched simulation entity data, is specifically used for: Based on the preset position of the dental chair, determine the reference point and force coordinate system of the force coordinate system; Obtain the pose information of the head model of the virtual patient in the force coordinate system; The force feedback tool performs diagnostic operations based on the head model's pose information in the force feedback coordinate system, generating force feedback information, which includes the pose information of the force feedback tool. Acquire the pose information of the virtual diagnostic tool in the force-sensing coordinate system; In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the bi-hand force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the matched reference coordinate system.

5. The system according to claim 4, characterized in that, When the virtual-real fusion head-mounted observer is used to generate the first virtual-real fusion diagnostic image based on the matched simulated entity data, it is specifically used for: Obtain the mapping relationship between the reference coordinate system and the device coordinate system; Based on the mapping relationship, the simulation entity data in the matched reference coordinate system is transferred to the device coordinate system to obtain the simulation entity data in the device coordinate system. The scene file is obtained, and a virtual image in the virtual-real fusion space is obtained based on the scene file and the simulation entity data in the device coordinate system. Acquire real-time images of the actual space captured in the photograph; The virtual image in the virtual-real fusion space is superimposed with the real image to obtain the first virtual-real fusion diagnostic image.

6. The system according to claim 1, characterized in that, The situation observation server is also used for: Displays virtual treatment scenarios corresponding to different treatment stations.

7. The system according to claim 1, characterized in that, The situation observation server is also used for: The system sets up a virtual patient call mode, which includes a normal call mode and an enhanced call mode. The virtual patients in the normal call mode have random causes of illness, while the virtual patients in the enhanced call mode have preset fixed causes of illness. Edit the virtual patient queue corresponding to the aforementioned queuing mode.

8. The system according to claim 1, characterized in that, The examination table also includes: a speaker and a microphone; Once the simulation computer loads a pre-created virtual patient in the virtual-real fusion space, the real user sends control commands to the virtual patient through the microphone and receives interactive voice output by the virtual patient through the speaker.

9. The system according to claim 1, characterized in that, The treatment table also includes: a height sensor and a pitch sensor; When the simulation computer detects a change in the height of the dental chair measured by the height sensor, or a change in the rotation angle of the dental chair measured by the pitch sensor, the simulation computer, based on the matching algorithm, controls the force sensing tool and the virtual diagnostic tool to re-match the poses of the two-hand force feedback tool to obtain new matched simulation entity data.

10. The system according to any one of claims 1-9, characterized in that, The oral treatment table training system based on spatial computing technology is connected to at least one external handheld terminal to display virtual-real fusion treatment images on the at least one external handheld terminal, and the at least one external terminal has augmented reality functionality.

11. A training method for oral diagnosis and treatment based on spatial computing technology, characterized in that, Oral diagnosis and treatment training systems applied to spatial computing technology include: The positioning data of physical objects and dental chairs existing in real space are obtained based on spatial computing technology, wherein the physical objects do not include the dental chairs. A scene file for oral diagnosis and treatment is generated based on the location data; Load the scene file and generate a virtual-real fusion space based on the scene file. In the virtual-real fusion space, load the pre-created virtual patient to be treated and the virtual medical tool corresponding to the real medical tool, and record the simulation entity data of the virtual patient and the virtual medical tool. A virtual diagnosis and treatment scenario is generated based on the scenario file and the simulation entity data. Force information is generated based on the diagnosis and treatment operation performed by a real user using a two-hand force feedback tool. The force information includes the pose information of the force feedback tool. Based on a matching algorithm, the pose of the force feedback tool and the virtual diagnosis and treatment tool is matched with that of the two-hand force feedback tool to obtain the matched simulation entity data. Based on the matched simulation entity data, a first virtual-real fusion diagnostic and treatment image is generated, which is used by real users to perform diagnostic and treatment operations. Obtain the simulation entity data corresponding to any treatment station; Based on the simulated entity data of any of the forwarded diagnostic and treatment stations, a second virtual-real fusion diagnostic and treatment image is generated; Both the first and second virtual-real fusion diagnostic images show a virtual patient closely positioned on a dental chair in the real space, receiving a diagnostic procedure.

12. The method according to claim 11, characterized in that, The acquisition of positioning data for physical objects and dental chairs existing in real space based on spatial computing technology includes: Scan the location images with significant features at a specified location in real space to determine the reference coordinate system and reference points; Obtain the pose information of the reference point in the current device coordinate system; Based on the pose information of the reference point in the current device coordinate system, establish a mapping relationship between the reference coordinate system and the device coordinate system; Measure the positioning data of physical objects existing in real space in the reference coordinate system; Scan a positioning image with significant features at the positioning reference point of the dental chair in real space to obtain the pose information of the dental chair in the device coordinate system; Based on the pose information of the dental chair in the device coordinate system and the mapping relationship, the positioning data of the dental chair in the reference coordinate system is obtained.

13. The method according to claim 12, characterized in that, The step of generating a scene file for oral diagnosis and treatment based on the positioning data includes: A virtual space is generated based on the positioning data of the entity object in the reference coordinate system; Based on the positioning data of the dental chair in the reference coordinate system, a virtual dental chair is generated at the corresponding position in the virtual space. In response to user actions, preset virtual facilities are generated within the virtual space; The coordinate and orientation data of the virtual space, the virtual dental chair, and the virtual facilities are obtained, and a scene file for oral diagnosis and treatment is generated based on the coordinate and orientation data.

14. The method according to claim 13, characterized in that, The process involves generating force information based on real users' diagnostic operations using the bi-handed force feedback tool. This force information includes the pose information of the force feedback tool. Based on a matching algorithm, the force feedback tool and the virtual diagnostic tool are matched with the pose of the bi-handed force feedback tool to obtain matched simulation entity data, including: Based on the preset position of the dental chair, determine the reference point and force coordinate system of the force coordinate system; Obtain the pose information of the head model of the virtual patient in the force coordinate system; The force feedback tool performs diagnostic operations based on the head model's pose information in the force feedback coordinate system, generating force feedback information, which includes the pose information of the force feedback tool. Acquire the pose information of the virtual diagnostic tool in the force-sensing coordinate system; In the force-sensing coordinate system, the pose information of the force-sensing tool and the virtual diagnostic tool in the force-sensing coordinate system is matched with the pose information of the bi-hand force feedback tool, and then transformed to the reference coordinate system to obtain the simulated entity data in the matched reference coordinate system.

15. The method according to claim 14, characterized in that, The step of generating a first virtual-real fusion diagnostic image based on the matched simulation entity data includes: Obtain the mapping relationship between the reference coordinate system and the device coordinate system; Based on the mapping relationship, the simulation entity data in the matched reference coordinate system is transferred to the device coordinate system to obtain the simulation entity data in the device coordinate system. The scene file is obtained, and a virtual image in the virtual-real fusion space is obtained based on the scene file and the simulation entity data in the device coordinate system. Acquire real-time images of the actual space captured in the photograph; The virtual image in the virtual-real fusion space is superimposed with the real image to obtain the first virtual-real fusion diagnostic image.

Images