1165 results about "Mixed reality" patented technology

The invention relates to the technical field of artificial intelligence assisted medical rehabilitation, in particular to a peripheral nerve injury personalized rehabilitation system and method based on a multi-modal large model, and the method comprises the steps: a feature fusion module employs space-time attention to fuse multi-modal time series data, and constructs an evaluation map; the personalized generation module is combined with historical data and reinforcement learning to generate a scheme containing virtual scene parameters; the interaction feedback module collects data through mixed reality and calculates action deviation; and the adaptive adjustment module adopts a meta-learning optimization model and a distributed iterative output scheme. According to the method, the cross-modal association precision of the motion features and the mechanical parameters is improved, dynamic matching of the training scene and the motion ability of the user is achieved, the virtual environment and the mechanical feedback threshold are optimized by dynamically adjusting the rehabilitation scheme parameters, the scheme optimization period is shortened based on an online iterative optimization mechanism, and the training efficiency is improved. The core defects of personalized adaptation lagging and low utilization efficiency of multi-modal data are overcome.

The invention discloses an electrical equipment operation state monitoring method, system and terminal, full life cycle health management of equipment is realized through multi-dimensional perception and intelligent analysis, a composite sensor network can be constructed from the level of the method, and high-frequency current, ultrahigh frequency, fiber grating temperature vibration, multi-parameter gas and acoustic sensors are integrated. Electromagnetic characteristics, mechanical states, environmental parameters and voiceprint characteristics are covered; the adaptive signal processing technology performs classification and noise reduction on multi-source data, and the three-dimensional digital twin model realizes time-space fusion of a temperature field, a vibration field, an electric field and a sound field; a lightweight space-time convolutional network is deployed to fuse a time domain waveform, a spectrogram and spatial distribution characteristics for diagnosis, and a hidden semi-Markov model and a particle filter algorithm are combined to dynamically predict the residual service life of equipment. The system architecture comprises a self-organizing sensor network with edge computing capability, a time-sensitive industrial communication network and a containerized analysis engine, and supports mixed reality visual interaction.

The invention relates to the technical field of data processing, and discloses a comprehensive management method and system for a smart park, and a storage medium. The method comprises the steps that environment, energy consumption and people flow data are collected through the Internet of Things, and a basic data set is obtained through edge node processing and semantic alignment; constructing a digital twin system by using a spatio-temporal data cube and a graphical mode; constructing a land digital identity system by adopting space-time coding and block chain technologies; performing anomaly identification and risk analysis on the activity data to obtain supervision early warning data; performing multi-dimensional analysis on the enterprise operation data to form an enterprise portrait; and presenting resource distribution through mixed reality, and analyzing cross-department data to obtain operation efficiency data. According to the invention, holographic perception, intelligent supervision and cooperative service of park elements are realized.

The invention relates to the field of man-machine interaction training, virtual reality and intelligent evaluation, and particularly discloses a psychological exercise ability training method and system based on mixed reality and behavior analysis. The method comprises the steps that a multi-scene psychological training environment is constructed and presented in mixed reality equipment, action behavior data, eye movement path data and operation response time data of a pilot in the training process are collected, multi-modal original behavior characteristics are constructed, and then a psychological exercise ability label set is constructed; and realizing capability-task modeling through matching of a capability label and a task structure, generating a personalized training path and a capability difference atlas based on difference analysis with a preset capability standard, adjusting task parameter configuration data, dynamically updating a training scene, generating a capability evolution trajectory, and outputting a training analysis result. In this way, accurate ability modeling, task dynamic adaptation and visual tracking of the training effect can be achieved, and the individuation and intelligence level of psychological exercise ability training is remarkably improved.

To provide an information processing system, a control method for the information processing system, and a program that allow a user to distinguish a sound from a specific sound source while leaving the sounds from each sound source as they are, even when a plurality of sound sources are present in a direction of interest that is of interest to the user.SOLUTION: An information processing system 1000 comprises: display means (a display unit 107) capable of displaying an image of a mixed reality space; playback means (an audio output unit 105) capable of playing back sounds from a plurality of sound sources to a user; vibration applying means (a vibration unit 117) for applying vibration to the user; control means (a control unit 111) for controlling displaying a virtual object that can be linked to each sound source on the display means; and selection means for selecting the virtual object (an object selection instruction unit 116). The vibration applying means applies vibration to the user in accordance with the intensity of the sound from the sound source associated with the virtual object if the virtual object is selected.SELECTED DRAWING: Figure 2

The invention relates to a nurse handover management system and method, and relates to the technical field of data processing, and the system comprises a real-time monitoring module which is used for constructing an individualized baseline database of a patient; the data integration module is used for outputting a multi-dimensional structured data set; the intelligent analysis engine is used for identifying clinical state change information of the patient and generating a corresponding risk early warning score; the decision support module is used for generating targeted nursing intervention suggestions; the interactive shift exchange platform is used for generating a structured shift exchange abstract based on the risk early warning score, the clinical change information and the nursing intervention suggestion, and dynamically displaying the structured shift exchange abstract through a mixed reality technology; and the closed-loop optimization module is used for receiving shift-change feedback information and subsequent execution data of the shift-change party, evaluating the completeness of shift-change information, updating a nursing plan, and optimizing and adjusting analysis and recommendation parameters of the system. The system can improve the accuracy and efficiency of nurse handover management.

In one embodiment, a method includes capturing images of a first user wearing a VR display device in a real-world environment. The method includes receiving a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes generating a first MR rendering of the first user in the VR environment., The first MR rendering of the first user is based on a compositing of the images of the first user and the VR rendering. The method includes receiving an indication of a user interaction with one or more elements of the VR environment in the first MR rendering. The method includes generating, in real-time responsive to the indication of the user interaction with the one or more elements, a second MR rendering of the first user in the VR environment. The one or more elements are modified according to the interaction.

Methods and system for atrial fibrillation ablations and other cardiac procedures utilizes a cardiac mapping system for navigation and guidance, and includes an additional virtual reality (VR) system and an augmented reality (AR) or mixed reality (MR) system. The Cardiac mapping system utilizes medical images and patient's electrical signals. The virtual reality (VR) system is utilized pre-procedure for visualization and pre-procedure planning. The augmented reality (AR) or mixed reality (MR) system is utilized pre-procedure or intra-procedure. Methods and system are disclosed utilizing pre-built virtual device models which interact with 3D volume rendered structures from patient's CT or MRI of the region of interest (ROI) for the different type of cardiac procedure. With virtual reality (VR) the operator utilizes hand-held sensors. With augmented reality (AR) or mixed reality (MR) a combination of hand gestures and an X-box controller is used to manipulate virtual devices, anatomical structures in the Hologram.

Particular embodiments are directed to passthrough image generation for a mixed-reality experience. A device may determine, using an eye-tracking system of a head-mounted device, eye-tracking data associated with a user of the head-mounted device. The device may determine, based on the eye-tracking data, a desired scene depth for the user. The device may instruct a first autofocus camera of the head-mounted device to adjust a first focus distance based on the desired scene depth and capture a first image of a real-world environment of the user. The device may generate a first passthrough image based on the first image. The device may display the first passthrough image to a first eye of the user via a first display of the head-mounted device.

A surgical planning system for use in surgical procedures to repair an anatomy of interest includes a preplanning system to generate a virtual surgical plan and a mixed reality system that includes a visualization device wearable by a user to view the virtual surgical plan projected in a real environment. The virtual surgical plan includes a 3D virtual model of the anatomy of interest. When wearing the visualization device, the user can align the 3D virtual model with the real anatomy of interest, thereby achieving a registration between details of the virtual surgical plan and the real anatomy of interest. The registration enables a surgeon to implement the virtual surgical plan on the real anatomy of interest without the use of tracking markers.

The invention relates to the field of training simulation, and discloses a public security AI-driven data generation type investigation teaching practical training system, which comprises the following modules: a case dynamic generation module, which is used for receiving original data of a public security case database, generating a logically closed virtual case through tensor decomposition, and sending the virtual case to the public security case database; transmitting the generated case data to a multi-modal virtual-real fusion training module; and the multi-modal virtual-real fusion training module is used for receiving the virtual case data, realizing interactive training of multi-modal clues through a mixed reality technology, dynamically injecting interference clues, and transmitting student operation data to the causal reinforcement learning evaluation module. Through the collaborative technical scheme of tensor decomposition and nonlinear constraint optimization, the integrity and space-time consistency generation of the virtual case evidence chain is realized, and compared with a traditional case generation method based on a fixed template, the problem of contradiction between material evidence association fracture and behavior logic is effectively solved.

The invention provides a low-altitude typical scene-oriented aircraft cockpit simulation method, system and device and a storage medium, and the method comprises the steps: constructing a dynamic extensible low-altitude scene library through fusing mixed reality (MR) and a high-precision digital twin model; in combination with a man-machine in-the-loop multimode interaction technology (eye movement-gesture-voice cooperation), the operation naturalness is improved; and an intelligent optimization algorithm driven by reinforcement learning is introduced, adaptive adjustment of flight control parameters is realized, and finally a'perception-decision-evaluation 'closed-loop training system is formed. According to the invention, digital twinning, mixed reality (MR), multi-modal sensing and adaptive control technologies are fused, pilot low-altitude flight skill training, man-machine work efficiency evaluation and intelligent aircraft control system verification are carried out, and the method can be expanded to UAM operator training and unmanned aircraft simulation testing. The technical barrier of low-altitude flight skill training at present is solved, and the technical blank of low-altitude dynamic scene simulation is filled up.

The invention is applicable to the technical field of medical image processing and mixed reality, and provides a laparoscopic surgery mixed reality navigation method based on deep learning and dynamic point tracking, which comprises the following steps: dynamically registering a three-dimensional model containing kidney, tumor and vessel with an initial frame of a laparoscope through a mixed reality alignment technology; the method comprises the following steps: constructing an operating forceps motion sensing model based on a time sequence deep neural network, realizing real-time control and parameter locking of a three-dimensional model pose, dynamically updating a two-dimensional feature point set by adopting a multi-feature-point combined tracker, constructing a candidate feature combination through a cross-quadrant sampling strategy, and constructing an operating forceps motion sensing model; a candidate feature combination is generated through a four-quadrant division and cross-regional sampling strategy, an optimal camera pose parameter is generated in combination with a re-projection error and pose continuity constraint, and an operation video is dynamically covered with a semitransparent three-dimensional model. The method can significantly enhance the spatial perception capability of the kidney anatomical structure, reduce the registration error of the kidney in the three-dimensional integrated kidney structure model and the laparoscope video, and improve the navigation precision.

A computer-implemented method for optimizing for virtual content occlusion in mixed reality may include (i) receiving a frame of a passthrough video captured by a sensor of a mixed-reality display, (ii) detecting at least one region of the frame of the passthrough video that is occluded by a virtual object projected within the mixed-reality display, (iii) creating a modified version of the frame of the passthrough video without the region that is occluded by the virtual object, (iv) providing the modified version of the frame to at least one visual analysis algorithm that calculates at least one statistic based on the modified version of the frame, (v) adjusting the frame by at least one visual adjustment algorithm based at least in part on the at least one statistic, and (vi) displaying the adjusted frame on the mixed-reality display. Various other methods, systems, and computer-readable media are also disclosed.

A device for simulating a weapon in a virtual reality (VR) environment, an augmented reality (AR) environment, a mixed reality (MR) environment, and / or combinations thereof. The device is adjustable to fit a user's shape and receive a gaming controller. The device includes a forearm grip and a front-end grip that improves the simulation of weaponry and user immersion.

The invention relates to a BIM model mobile terminal and mixed reality cooperation method and system and a medium, and the method comprises the steps: carrying out the space segmentation and space grid division processing of a BIM model according to a pre-created index structure, obtaining the index information of a plurality of space models and a plurality of space grids, and obtaining a plurality of space models and a plurality of space grids based on the index information; and constructing a unique identification code of the spatial model. Identifying the identification code through the mobile terminal, loading the spatial model corresponding to the identification code, obtaining spatial coordinates in the spatial model according to the gesture operation, converting the spatial coordinates into spatial grid index information through a preset conversion rule, and generating a loading instruction of the mixed reality equipment. And the mixed reality equipment loads a space model corresponding to the space grid index information according to the loading instruction, and superposes the space model to a real building scene through a positioning strategy. The problem of low cooperation efficiency of a BIM model mobile terminal and mixed reality in related technologies is solved.

In some embodiments, an electronic device presents an M R environment including real content and / or virtual content. In some embodiments, a client application provides an API with a target region of the MR environment, one or more criteria, and a presentation effect. In response to the one or more criteria being satisfied, the electronic device presents the target region of the MR environment with the presentation effect.

The invention discloses a cultural relic three-dimensional dynamic scene display system based on a virtual-real fusion technology, and relates to the technical field of mixed reality, and the system comprises a data collection module which obtains original signals of a to-be-displayed cultural relic and a surrounding environment, and generates an environment feature vector and an environment fusion index; the twinborn simulation module is used for generating health state feature vectors of the cultural relics in combination with finite element grids and carrying out prediction simulation on the deterioration state of cultural relic materials; the model creating module is used for creating a three-tense model resource package and generating time axis metadata by adopting a vertex interpolation and mapping mixing technology; the story layout module is used for constructing an environment event mapping rule base, obtaining a ranked trigger event list through priority ranking, and generating a scene script sequence; according to the method, the health state and the degradation process of the cultural relics under different environmental conditions can be dynamically reflected, and three-dimensional scene display changing along with time evolution is provided.

Disclosed are approaches that may provide image-guidance to interventionalists by providing true 3D visualization and quantitative feedback in real-time. A guidance system may allow a physician to manipulate a medical device and see a 3D rendering with quantitative feedback floating in mixed reality, next to standard monitors. Image tracking may detect and co-register the medical device's 3D position using, for example, bi-plane C-arm X-ray fluoroscopy and provide a 3D trajectory as quantitative feedback. Patterns in a fluoroscopic image may be used to accurately determine an object's z-position from a single angle projection.

The invention relates to the technical field of intelligent education, in particular to an AI-based nursing teaching scene construction and evaluation system. According to the system, a dynamic case script is generated through historical case data, and a virtual patient physiological model is constructed; creating three teaching scenes of community follow-up visit, emergency rescue and traditional Chinese medicine nursing fusion by using a mixed reality technology; a personalized learning portrait of the student is constructed through a pre-test, and accurate matching between the ability of the student and the teaching content is formed; a double-track system skill training module is built, and AI tutor guidance and teacher guidance are integrated; and an ability evaluation system based on operation data is established, and intelligent pushing of adaptive training content is realized. The problems that in traditional nursing teaching, scene authenticity is insufficient, personalized training is insufficient, and clinical thinking is difficult to cultivate are solved, and the clinical operation skill and decision judgment ability of students are improved.

The invention provides a mixed reality industrial simulation training system which is composed of a space fusion perception unit, a multi-physics field simulation unit, a cooperative training management unit and a self-adaptive teaching engine. The space fusion sensing unit is composed of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm, and the real-time environment modeling and positioning subsystem dynamically constructs a three-dimensional space coordinate system of a real industrial scene through a visual sensor array and a radio frequency scanning device; generating a pose matrix of the virtual equipment model in the perspective display interface; the physical interaction detection algorithm realizes contact force feedback between virtual and real objects based on dynamic solution of collision volume, and when an operator triggers a virtual switch through a handheld controller, a real equipment control signal is synchronously driven; the multi-physics field simulation unit comprises a physical engine core module and a fault injection interface.

The invention discloses a lithium battery equipment temperature equalization and energy consumption optimization system and method based on digital twinning. Constructing a digital twinborn model of the lithium battery equipment, and realizing fusion interaction of multi-physical parameter data; establishing a balanced energy-saving control model between the temperature field of the internal space of the cavity and the temperature control parameters through an ANFIS technology; the turbulent flow wind speed optimization module is used for cooperatively regulating and controlling the rotating speed of a fan and the temperature of a cavity by adopting a double-closed-loop feedback control strategy; an equipment health state evaluation model is constructed based on a deep learning algorithm, and fault early warning and optimization suggestion automatic generation are realized; a mixed reality technology is innovatively introduced in a human-computer interaction level, a multi-mode interaction mode is provided, and the ability of an operator to understand and control the running state of equipment is improved in combination with a mixed reality display function. According to the invention, real-time monitoring and optimal control of the running state of the lithium battery equipment are realized, the balance of the temperature field and the energy efficiency level of the system are remarkably improved, and important technical support is provided for green intelligent manufacturing.

Mixed reality instructional technology is enhanced. Comparison of a guide mixed reality data stream to a candidate mixed reality data stream detects a deviation, e.g., different hand movement, different tool placement, or different sensor telemetry. Experiencing a feedback stream of output that is based on the deviation facilitates device control and helps candidates improve their skills. Some feedback output emphasizes deviation size, e.g., by varying colors, sounds, or haptic output proportionally to the deviation size. Some feedback output renders a translucent or skeletal guide overlaid on a live video of current candidate activity. Some embodiments support searches whose results show, e.g., how a certain expert performed a task differently than the candidate, or examples of a certain task such as widget replacement in the field. Stream optimization, summarization, synchronization, and other stream derivation functionality is provided.

The invention relates to the technical field of mixed reality and artificial intelligence fusion application, in particular to a mixed reality intelligent inspection method and system based on multi-modal fusion. According to the technical scheme, the method comprises the following steps: synchronously acquiring a user view image and voice stream data through mixed reality equipment; performing target detection on the view image by adopting an improved YOLOv10 model, and outputting a candidate target area and a confidence score # imgabs0 # thereof; voice input is transferred through a Whisper voice recognition model, and a semantic vector # imgabs1 # is generated by using a BERT semantic embedding model. According to the invention, through a multi-mode cooperative sensing mechanism of fusing image, voice and space gazing information, in combination with context semantic optimization and lightweight terminal deployment, intelligent target identification and natural interaction with high precision, real-time performance and strong robustness are realized in a complex industrial inspection scene.

A Mixed Reality (MR) and Artificial Intelligence (AI)-enhanced fire investigation system that analyzes data descriptive of fire-damaged locations, identifies objects of the location, creates a 3-D model of the location, automatically analyzes the location utilizing an AI fire investigation model, automatically analyzes the location utilizing an AI safety evaluation model, and automatically embeds and provides layered access to data assigned to the 3-D model.

Disclosed herein are systems and methods for distributed computing and / or networking for mixed reality systems. A method may include capturing an image via a camera of a head-wearable device. Inertial data may be captured via an inertial measurement unit of the head-wearable device. A position of the head-wearable device can be estimated based on the image and the inertial data via one or more processors of the head-wearable device. The image can be transmitted to a remote server. A neural network can be trained based on the image via the remote server. A trained neural network can be transmitted to the head-wearable device.

The utilization of drones, mixed reality displays, novel networking techniques, advanced networking techniques (e.g., edge computing), and AI-driven data fusion and visualization. This combination of technologies is employed in tandem to provide advanced data visualization capabilities in mixed reality displays, primarily for usage between multiple devices including mixed reality displays, drones, and other systems that would support networking and data collection capabilities. These capabilities are primarily intended for usage between multiple mixed reality displays, although could be applied to robots, autonomous vehicles, guidance systems, weapon systems, and detection systems.

A system and method for 3D anomaly detection and tracking are provided that uses multimodal fusion, reduced training data, recursive segmentation and histogram statistic distance. The anomaly may be a defect or a configuration error.