1082 results about "Mixed reality" patented technology

The invention relates to the technical field of artificial intelligence assisted medical rehabilitation, in particular to a peripheral nerve injury personalized rehabilitation system and method based on a multi-modal large model, and the method comprises the steps: a feature fusion module employs space-time attention to fuse multi-modal time series data, and constructs an evaluation map; the personalized generation module is combined with historical data and reinforcement learning to generate a scheme containing virtual scene parameters; the interaction feedback module collects data through mixed reality and calculates action deviation; and the adaptive adjustment module adopts a meta-learning optimization model and a distributed iterative output scheme. According to the method, the cross-modal association precision of the motion features and the mechanical parameters is improved, dynamic matching of the training scene and the motion ability of the user is achieved, the virtual environment and the mechanical feedback threshold are optimized by dynamically adjusting the rehabilitation scheme parameters, the scheme optimization period is shortened based on an online iterative optimization mechanism, and the training efficiency is improved. The core defects of personalized adaptation lagging and low utilization efficiency of multi-modal data are overcome.

The invention discloses an electrical equipment operation state monitoring method, system and terminal, full life cycle health management of equipment is realized through multi-dimensional perception and intelligent analysis, a composite sensor network can be constructed from the level of the method, and high-frequency current, ultrahigh frequency, fiber grating temperature vibration, multi-parameter gas and acoustic sensors are integrated. Electromagnetic characteristics, mechanical states, environmental parameters and voiceprint characteristics are covered; the adaptive signal processing technology performs classification and noise reduction on multi-source data, and the three-dimensional digital twin model realizes time-space fusion of a temperature field, a vibration field, an electric field and a sound field; a lightweight space-time convolutional network is deployed to fuse a time domain waveform, a spectrogram and spatial distribution characteristics for diagnosis, and a hidden semi-Markov model and a particle filter algorithm are combined to dynamically predict the residual service life of equipment. The system architecture comprises a self-organizing sensor network with edge computing capability, a time-sensitive industrial communication network and a containerized analysis engine, and supports mixed reality visual interaction.

The invention relates to a nurse handover management system and method, and relates to the technical field of data processing, and the system comprises a real-time monitoring module which is used for constructing an individualized baseline database of a patient; the data integration module is used for outputting a multi-dimensional structured data set; the intelligent analysis engine is used for identifying clinical state change information of the patient and generating a corresponding risk early warning score; the decision support module is used for generating targeted nursing intervention suggestions; the interactive shift exchange platform is used for generating a structured shift exchange abstract based on the risk early warning score, the clinical change information and the nursing intervention suggestion, and dynamically displaying the structured shift exchange abstract through a mixed reality technology; and the closed-loop optimization module is used for receiving shift-change feedback information and subsequent execution data of the shift-change party, evaluating the completeness of shift-change information, updating a nursing plan, and optimizing and adjusting analysis and recommendation parameters of the system. The system can improve the accuracy and efficiency of nurse handover management.

Particular embodiments are directed to passthrough image generation for a mixed-reality experience. A device may determine, using an eye-tracking system of a head-mounted device, eye-tracking data associated with a user of the head-mounted device. The device may determine, based on the eye-tracking data, a desired scene depth for the user. The device may instruct a first autofocus camera of the head-mounted device to adjust a first focus distance based on the desired scene depth and capture a first image of a real-world environment of the user. The device may generate a first passthrough image based on the first image. The device may display the first passthrough image to a first eye of the user via a first display of the head-mounted device.

A surgical planning system for use in surgical procedures to repair an anatomy of interest includes a preplanning system to generate a virtual surgical plan and a mixed reality system that includes a visualization device wearable by a user to view the virtual surgical plan projected in a real environment. The virtual surgical plan includes a 3D virtual model of the anatomy of interest. When wearing the visualization device, the user can align the 3D virtual model with the real anatomy of interest, thereby achieving a registration between details of the virtual surgical plan and the real anatomy of interest. The registration enables a surgeon to implement the virtual surgical plan on the real anatomy of interest without the use of tracking markers.

The invention relates to the field of training simulation, and discloses a public security AI-driven data generation type investigation teaching practical training system, which comprises the following modules: a case dynamic generation module, which is used for receiving original data of a public security case database, generating a logically closed virtual case through tensor decomposition, and sending the virtual case to the public security case database; transmitting the generated case data to a multi-modal virtual-real fusion training module; and the multi-modal virtual-real fusion training module is used for receiving the virtual case data, realizing interactive training of multi-modal clues through a mixed reality technology, dynamically injecting interference clues, and transmitting student operation data to the causal reinforcement learning evaluation module. Through the collaborative technical scheme of tensor decomposition and nonlinear constraint optimization, the integrity and space-time consistency generation of the virtual case evidence chain is realized, and compared with a traditional case generation method based on a fixed template, the problem of contradiction between material evidence association fracture and behavior logic is effectively solved.

The invention provides a low-altitude typical scene-oriented aircraft cockpit simulation method, system and device and a storage medium, and the method comprises the steps: constructing a dynamic extensible low-altitude scene library through fusing mixed reality (MR) and a high-precision digital twin model; in combination with a man-machine in-the-loop multimode interaction technology (eye movement-gesture-voice cooperation), the operation naturalness is improved; and an intelligent optimization algorithm driven by reinforcement learning is introduced, adaptive adjustment of flight control parameters is realized, and finally a'perception-decision-evaluation 'closed-loop training system is formed. According to the invention, digital twinning, mixed reality (MR), multi-modal sensing and adaptive control technologies are fused, pilot low-altitude flight skill training, man-machine work efficiency evaluation and intelligent aircraft control system verification are carried out, and the method can be expanded to UAM operator training and unmanned aircraft simulation testing. The technical barrier of low-altitude flight skill training at present is solved, and the technical blank of low-altitude dynamic scene simulation is filled up.

The invention is applicable to the technical field of medical image processing and mixed reality, and provides a laparoscopic surgery mixed reality navigation method based on deep learning and dynamic point tracking, which comprises the following steps: dynamically registering a three-dimensional model containing kidney, tumor and vessel with an initial frame of a laparoscope through a mixed reality alignment technology; the method comprises the following steps: constructing an operating forceps motion sensing model based on a time sequence deep neural network, realizing real-time control and parameter locking of a three-dimensional model pose, dynamically updating a two-dimensional feature point set by adopting a multi-feature-point combined tracker, constructing a candidate feature combination through a cross-quadrant sampling strategy, and constructing an operating forceps motion sensing model; a candidate feature combination is generated through a four-quadrant division and cross-regional sampling strategy, an optimal camera pose parameter is generated in combination with a re-projection error and pose continuity constraint, and an operation video is dynamically covered with a semitransparent three-dimensional model. The method can significantly enhance the spatial perception capability of the kidney anatomical structure, reduce the registration error of the kidney in the three-dimensional integrated kidney structure model and the laparoscope video, and improve the navigation precision.

A computer-implemented method for optimizing for virtual content occlusion in mixed reality may include (i) receiving a frame of a passthrough video captured by a sensor of a mixed-reality display, (ii) detecting at least one region of the frame of the passthrough video that is occluded by a virtual object projected within the mixed-reality display, (iii) creating a modified version of the frame of the passthrough video without the region that is occluded by the virtual object, (iv) providing the modified version of the frame to at least one visual analysis algorithm that calculates at least one statistic based on the modified version of the frame, (v) adjusting the frame by at least one visual adjustment algorithm based at least in part on the at least one statistic, and (vi) displaying the adjusted frame on the mixed-reality display. Various other methods, systems, and computer-readable media are also disclosed.

A device for simulating a weapon in a virtual reality (VR) environment, an augmented reality (AR) environment, a mixed reality (MR) environment, and / or combinations thereof. The device is adjustable to fit a user's shape and receive a gaming controller. The device includes a forearm grip and a front-end grip that improves the simulation of weaponry and user immersion.

The invention relates to a BIM model mobile terminal and mixed reality cooperation method and system and a medium, and the method comprises the steps: carrying out the space segmentation and space grid division processing of a BIM model according to a pre-created index structure, obtaining the index information of a plurality of space models and a plurality of space grids, and obtaining a plurality of space models and a plurality of space grids based on the index information; and constructing a unique identification code of the spatial model. Identifying the identification code through the mobile terminal, loading the spatial model corresponding to the identification code, obtaining spatial coordinates in the spatial model according to the gesture operation, converting the spatial coordinates into spatial grid index information through a preset conversion rule, and generating a loading instruction of the mixed reality equipment. And the mixed reality equipment loads a space model corresponding to the space grid index information according to the loading instruction, and superposes the space model to a real building scene through a positioning strategy. The problem of low cooperation efficiency of a BIM model mobile terminal and mixed reality in related technologies is solved.

In some embodiments, an electronic device presents an M R environment including real content and / or virtual content. In some embodiments, a client application provides an API with a target region of the MR environment, one or more criteria, and a presentation effect. In response to the one or more criteria being satisfied, the electronic device presents the target region of the MR environment with the presentation effect.

The invention discloses a cultural relic three-dimensional dynamic scene display system based on a virtual-real fusion technology, and relates to the technical field of mixed reality, and the system comprises a data collection module which obtains original signals of a to-be-displayed cultural relic and a surrounding environment, and generates an environment feature vector and an environment fusion index; the twinborn simulation module is used for generating health state feature vectors of the cultural relics in combination with finite element grids and carrying out prediction simulation on the deterioration state of cultural relic materials; the model creating module is used for creating a three-tense model resource package and generating time axis metadata by adopting a vertex interpolation and mapping mixing technology; the story layout module is used for constructing an environment event mapping rule base, obtaining a ranked trigger event list through priority ranking, and generating a scene script sequence; according to the method, the health state and the degradation process of the cultural relics under different environmental conditions can be dynamically reflected, and three-dimensional scene display changing along with time evolution is provided.

Disclosed are approaches that may provide image-guidance to interventionalists by providing true 3D visualization and quantitative feedback in real-time. A guidance system may allow a physician to manipulate a medical device and see a 3D rendering with quantitative feedback floating in mixed reality, next to standard monitors. Image tracking may detect and co-register the medical device's 3D position using, for example, bi-plane C-arm X-ray fluoroscopy and provide a 3D trajectory as quantitative feedback. Patterns in a fluoroscopic image may be used to accurately determine an object's z-position from a single angle projection.

The invention relates to the technical field of intelligent education, in particular to an AI-based nursing teaching scene construction and evaluation system. According to the system, a dynamic case script is generated through historical case data, and a virtual patient physiological model is constructed; creating three teaching scenes of community follow-up visit, emergency rescue and traditional Chinese medicine nursing fusion by using a mixed reality technology; a personalized learning portrait of the student is constructed through a pre-test, and accurate matching between the ability of the student and the teaching content is formed; a double-track system skill training module is built, and AI tutor guidance and teacher guidance are integrated; and an ability evaluation system based on operation data is established, and intelligent pushing of adaptive training content is realized. The problems that in traditional nursing teaching, scene authenticity is insufficient, personalized training is insufficient, and clinical thinking is difficult to cultivate are solved, and the clinical operation skill and decision judgment ability of students are improved.

The invention provides a mixed reality industrial simulation training system which is composed of a space fusion perception unit, a multi-physics field simulation unit, a cooperative training management unit and a self-adaptive teaching engine. The space fusion sensing unit is composed of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm, and the real-time environment modeling and positioning subsystem dynamically constructs a three-dimensional space coordinate system of a real industrial scene through a visual sensor array and a radio frequency scanning device; generating a pose matrix of the virtual equipment model in the perspective display interface; the physical interaction detection algorithm realizes contact force feedback between virtual and real objects based on dynamic solution of collision volume, and when an operator triggers a virtual switch through a handheld controller, a real equipment control signal is synchronously driven; the multi-physics field simulation unit comprises a physical engine core module and a fault injection interface.

The invention discloses a lithium battery equipment temperature equalization and energy consumption optimization system and method based on digital twinning. Constructing a digital twinborn model of the lithium battery equipment, and realizing fusion interaction of multi-physical parameter data; establishing a balanced energy-saving control model between the temperature field of the internal space of the cavity and the temperature control parameters through an ANFIS technology; the turbulent flow wind speed optimization module is used for cooperatively regulating and controlling the rotating speed of a fan and the temperature of a cavity by adopting a double-closed-loop feedback control strategy; an equipment health state evaluation model is constructed based on a deep learning algorithm, and fault early warning and optimization suggestion automatic generation are realized; a mixed reality technology is innovatively introduced in a human-computer interaction level, a multi-mode interaction mode is provided, and the ability of an operator to understand and control the running state of equipment is improved in combination with a mixed reality display function. According to the invention, real-time monitoring and optimal control of the running state of the lithium battery equipment are realized, the balance of the temperature field and the energy efficiency level of the system are remarkably improved, and important technical support is provided for green intelligent manufacturing.

The invention relates to the technical field of mixed reality and artificial intelligence fusion application, in particular to a mixed reality intelligent inspection method and system based on multi-modal fusion. According to the technical scheme, the method comprises the following steps: synchronously acquiring a user view image and voice stream data through mixed reality equipment; performing target detection on the view image by adopting an improved YOLOv10 model, and outputting a candidate target area and a confidence score # imgabs0 # thereof; voice input is transferred through a Whisper voice recognition model, and a semantic vector # imgabs1 # is generated by using a BERT semantic embedding model. According to the invention, through a multi-mode cooperative sensing mechanism of fusing image, voice and space gazing information, in combination with context semantic optimization and lightweight terminal deployment, intelligent target identification and natural interaction with high precision, real-time performance and strong robustness are realized in a complex industrial inspection scene.

A Mixed Reality (MR) and Artificial Intelligence (AI)-enhanced fire investigation system that analyzes data descriptive of fire-damaged locations, identifies objects of the location, creates a 3-D model of the location, automatically analyzes the location utilizing an AI fire investigation model, automatically analyzes the location utilizing an AI safety evaluation model, and automatically embeds and provides layered access to data assigned to the 3-D model.

Disclosed herein are systems and methods for distributed computing and / or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.

The utilization of drones, mixed reality displays, novel networking techniques, advanced networking techniques (e.g., edge computing), and AI-driven data fusion and visualization. This combination of technologies is employed in tandem to provide advanced data visualization capabilities in mixed reality displays, primarily for usage between multiple devices including mixed reality displays, drones, and other systems that would support networking and data collection capabilities. These capabilities are primarily intended for usage between multiple mixed reality displays, although could be applied to robots, autonomous vehicles, guidance systems, weapon systems, and detection systems.

A system and method for 3D anomaly detection and tracking are provided that uses multimodal fusion, reduced training data, recursive segmentation and histogram statistic distance. The anomaly may be a defect or a configuration error.

The invention relates to the technical field of 3D printing control, and discloses a 3D remote printing control system and method based on the Internet of Things, and the 3D remote printing control method based on the Internet of Things comprises the steps: collecting multi-dimensional data in a printing process through a multi-modal sensor network; constructing a digital twinborn body of a printing environment by adopting a mixed reality technology; constructing a tactile signal model and a sound feature model, and cooperatively transmitting multi-sensory feedback matched with a real printing process to a user with the digital twinborn body; calculating the cognitive load level of the user in real time, and dynamically adjusting the information presentation density of each sensory channel according to the load level; the system converts the user intention into a printing equipment control command according to the user cognitive load state and the printing state information flow, and provides execution confirmation information through multi-sensory feedback; and the control experience is adaptively optimized according to printing task characteristics and user preferences, so that the remote operation effect is close to that of field operation.

This invention provides an image processor, a processing method, a storage medium, and a mixed reality display system. The image processor includes: a first color management module, whose input is connected to a rendering module, for acquiring virtual layer data from the rendering module and performing first color processing on it; a second color management module, whose input is connected to a camera module, for acquiring real-world layer data from the camera module and performing second color processing on it; a layer blending module, whose inputs are respectively connected to the first and second color management modules, for performing layer blending processing on the color-processed virtual layer data and real-world layer data to determine mixed image data; and a third color management module, whose input is connected to the layer blending module, for acquiring mixed image data from the layer blending module, performing third color processing on it, and outputting the third-color-processed mixed image data to a mixed reality display.

The invention discloses a mixed reality mine ventilation equipment maintenance guidance method. The method comprises the following steps: S1, collecting real-time state data of mine ventilation equipment based on a multi-source sensor; s2, collecting historical maintenance data, performing natural language preprocessing on the historical maintenance data, constructing an entity-relation-attribute triple for the preprocessed data through a BERT model, and obtaining an optimized maintenance knowledge graph based on the entity-relation-attribute triple; s3, taking the maintenance process of the mine equipment as a Markov decision process, and taking the optimized maintenance knowledge graph as a state space to carry out Markov decision process modeling to obtain an MDP prediction model; and S4, based on the MDP prediction model, constructing intelligent agent learning through improved deep reinforcement learning, selecting an optimal maintenance action strategy for different equipment states under the MDP prediction model, and outputting an optimal maintenance guidance strategy.

A system for registering position and orientation (P&O) of mobile devices relative to a construction site may include: a light projector; at least one sensor configured to record at least one set of XYZ coordinates of at least one specific surface, and a set of XYZ coordinates of the light projector within the construction site; and a computer processor, wherein the light projector is configured to project the XYZ coordinates of the at least one set of position markers onto the at least one specific surface; and wherein the projection of the XYZ coordinates of the at least one set of position markers is detectable by at least one sensor located at a mobile device, allowing registration of XYZ coordinates of a virtual reality interface of the mobile device relative to the projection of the position markers onto the at least one specific surface.

The invention provides a wearable tactile feedback adjusting device used for simulating material surface texture characteristics in real time in a mixed reality environment. The invention relates to a method. The device comprises a U-shaped pneumatic actuator surrounding the fingertip of a user, a fingerstall support for accommodating the fingertip of the user, a resonance actuator located on the fingerstall support and capable of generating vibrotactile feedback to enhance the perceived roughness when the user touches the surface of an object, and a pneumatic control system that can generate a pneumatic drive command to reduce roughness perception when touching the surface of the object. Notably, the fingertip and the finger pulp of the user can still directly touch the surface of the object and cannot be blocked by the U-shaped pneumatic actuator, so that the real texture simulation experience is realized.

The invention discloses a human body meridian point visualization method based on intelligent AI and a visualization teaching system, and relates to the technical field of human body meridian points. According to the human body meridian point visualization method based on the intelligent AI, a three-dimensional virtual human body model is constructed based on magnetic resonance imaging and computed tomography; acupoint positioning and meridian drawing are carried out on the constructed three-dimensional virtual human body model, and visualization is carried out; relative spatial position relations of human meridians, acupuncture points and human tissues are displayed based on a quantitative mixed reality technology, effective integration of advantage information of two images is realized on an image level, a foundation is laid for obtaining comprehensive and accurate human tissue and organ structure data, and a model is established on the aspect of modeling. The problem that a complex human body structure and a space relation are difficult to present in traditional modeling is solved, accurate labeling and visual display of meridian points on the virtual model are achieved in meridian point processing, and in a teaching system, interaction convenience and naturalness of a user and the model are improved.