624 results about "Mixed reality" patented technology

In some embodiments, an electronic device presents an M R environment including real content and / or virtual content. In some embodiments, a client application provides an API with a target region of the MR environment, one or more criteria, and a presentation effect. In response to the one or more criteria being satisfied, the electronic device presents the target region of the MR environment with the presentation effect.

The invention discloses a cultural relic three-dimensional dynamic scene display system based on a virtual-real fusion technology, and relates to the technical field of mixed reality, and the system comprises a data collection module which obtains original signals of a to-be-displayed cultural relic and a surrounding environment, and generates an environment feature vector and an environment fusion index; the twinborn simulation module is used for generating health state feature vectors of the cultural relics in combination with finite element grids and carrying out prediction simulation on the deterioration state of cultural relic materials; the model creating module is used for creating a three-tense model resource package and generating time axis metadata by adopting a vertex interpolation and mapping mixing technology; the story layout module is used for constructing an environment event mapping rule base, obtaining a ranked trigger event list through priority ranking, and generating a scene script sequence; according to the method, the health state and the degradation process of the cultural relics under different environmental conditions can be dynamically reflected, and three-dimensional scene display changing along with time evolution is provided.

Disclosed are approaches that may provide image-guidance to interventionalists by providing true 3D visualization and quantitative feedback in real-time. A guidance system may allow a physician to manipulate a medical device and see a 3D rendering with quantitative feedback floating in mixed reality, next to standard monitors. Image tracking may detect and co-register the medical device's 3D position using, for example, bi-plane C-arm X-ray fluoroscopy and provide a 3D trajectory as quantitative feedback. Patterns in a fluoroscopic image may be used to accurately determine an object's z-position from a single angle projection.

The invention relates to the technical field of intelligent education, in particular to an AI-based nursing teaching scene construction and evaluation system. According to the system, a dynamic case script is generated through historical case data, and a virtual patient physiological model is constructed; creating three teaching scenes of community follow-up visit, emergency rescue and traditional Chinese medicine nursing fusion by using a mixed reality technology; a personalized learning portrait of the student is constructed through a pre-test, and accurate matching between the ability of the student and the teaching content is formed; a double-track system skill training module is built, and AI tutor guidance and teacher guidance are integrated; and an ability evaluation system based on operation data is established, and intelligent pushing of adaptive training content is realized. The problems that in traditional nursing teaching, scene authenticity is insufficient, personalized training is insufficient, and clinical thinking is difficult to cultivate are solved, and the clinical operation skill and decision judgment ability of students are improved.

A system and method for 3D anomaly detection and tracking are provided that uses multimodal fusion, reduced training data, recursive segmentation and histogram statistic distance. The anomaly may be a defect or a configuration error.

This invention provides an image processor, a processing method, a storage medium, and a mixed reality display system. The image processor includes: a first color management module, whose input is connected to a rendering module, for acquiring virtual layer data from the rendering module and performing first color processing on it; a second color management module, whose input is connected to a camera module, for acquiring real-world layer data from the camera module and performing second color processing on it; a layer blending module, whose inputs are respectively connected to the first and second color management modules, for performing layer blending processing on the color-processed virtual layer data and real-world layer data to determine mixed image data; and a third color management module, whose input is connected to the layer blending module, for acquiring mixed image data from the layer blending module, performing third color processing on it, and outputting the third-color-processed mixed image data to a mixed reality display.

According to one embodiment, a mixed reality device overlays, on a real space, and displays a virtual object in a virtual space. The mixed reality device is configured to obtain a position of an object in the real space and a position of the virtual object. The mixed reality device is configured to change an external appearance of the virtual object when the virtual object is positioned on a far side of the object relative to the mixed reality device and overlaps the object.

The invention discloses a building equipment and material intelligent transportation analysis system based on mixed reality, and relates to the technical field of transportation in the construction process, and the system comprises a data collection module, a virtual model construction module, a transportation path planning module, an equipment disassembly degree measurement and calculation module, a scheme economy evaluation module and an optimal scheme output module. According to the invention, on the basis of mixed reality equipment and an unmanned aerial vehicle telemetering technology, high-efficiency acquisition and high-precision three-dimensional virtual modeling of transportation channel and equipment data are completed, manual measurement limitation is broken through, traffic is dynamically simulated, paths are screened, calculation and disassembly are carried out, the structure safety is guaranteed, and a complex scene is adapted; a cost-benefit balance algorithm is introduced, a multi-dimensional evaluation system is constructed, the cost and influence are comprehensively considered, additional evaluation is dynamically triggered, a special scene is adjustable, and scientific support is provided for intelligent transportation management.

The invention belongs to the technical field of medical training, and particularly relates to a method and a system for constructing a virtual operating room resource library for reappearing an offshore operation scene, and the method comprises the following steps: 1, collecting offshore medical data and scene demands: collecting an offshore environment and medical data, and determining medical care core capability demands; in the application, coupling parameters generated by a sea condition dynamic sensing module drive a mixed reality scene rendering module to present real shake and environment change; meanwhile, parameters are synchronously input to the equipment modeling module, fault response of virtual equipment is triggered, a user operates the virtual equipment through the interaction module and executes an operation on a virtual human body model, and the resource library management module guarantees reusability and updatability of resources in the whole process. Through real-time data linkage among the modules, total-factor dynamic coupling of sea conditions, scenes, equipment and operation objects is finally realized, and the problem of simulation distortion of a traditional system is solved.

An optical system is presented for use in a near-eye mixed reality system. The system comprises a relay system defining an eyebox of the optical system, said relay system being configured and operable to relay a virtual image light field from a light-engine onto an eye pupil plane while combining said virtual image with real-world light field. The relay system is configured as a free space relay system configured for free space propagation of said virtual image light field being relayed, said free space relay system comprising at least one off-axis 4f-system. Each of said at least one off-axis 4f-system comprises at least one lens formed from at least one resonance-domain surface relief diffractive optical element (SRDOE) operable for combining said virtual image light field with the real-world light field, said at least one SRDOE being configured with a predetermined global surface relief pattern characterized by global variation of at least some of pattern parameters across said SRDOE.

The invention relates to the cross technical field of medical education and mixed reality technology, in particular to a traditional Chinese medicine meridian point visualization system based on mixed reality technology, which comprises a data fusion and personalized calibration module, a virtual-real fusion interaction module, an interaction teaching module, an intelligent evaluation module and a meridian qi and blood simulation module. By fusing medical image data, depth sensing data and a traditional Chinese medicine acupoint selection rule, a personalized three-dimensional human body model is constructed, and acupoints are accurately calibrated; real-time registration and shielding processing of the virtual model and a real scene are achieved through mixed reality equipment; various natural interaction modes such as gestures, voices and sight lines are supported, and an integrated function of learning, training and assessment is provided; and the meridian qi-blood running state can be dynamically simulated based on the midnight-noon ebb-flow theory. The system has the advantages that the problems that traditional Chinese medicine teaching is poor in intuition and interactivity, cannot be personalized, is inconvenient to examine and the like are solved, and immersive, interactive and intelligent teaching of traditional Chinese medicine meridian point knowledge is achieved.

Described herein are an apparatus and method for providing mixed reality content. The apparatus for providing mixed reality content includes: memory configured to store a program and data required for providing mixed reality content; and a controller provided with at least one processor, and configured to operate by executing the program stored in the memory, to receive a captured frame image, and to display a virtual scene, including a plurality of virtual objects generated based on the results of analyzing the frame image, on a display. The controller performs operations required for analyzing the frame image and generating a virtual object on a per-virtual object basis, updates a generated virtual object in the virtual scene regardless of whether generation of another virtual object is completed, and renders the virtual scene and displays it on the display in accordance with a display cycle.

The present disclosure is directed towards systems and methods for interacting with a computing device in a virtual, augmented, or mixed reality environment. The system comprises a ring input device worn on a user's middle phalange on the middle finger and may also comprise a tracker module for tracking index finger pointing. The ring input device has a touch-sensitive surface for receiving input from the user's thumb, without substantially bending the thumb. The computing device is operably coupled to both the ring input device and the tracker module and configured to receive input from both simultaneously and execute actions corresponding to the received inputs. The disclosure offers an intuitive and effective approach to interacting with computing devices in various environments.

The invention relates to the technical field of robots, in particular to a mobile robot remote operation system and method based on mixed reality, and the method comprises an operator end and a robot end, the operator end completes a complex remote operation task based on visual feedback provided by a man-machine cooperation interaction interface, and the robot end completes a remote operation task based on visual feedback provided by the man-machine cooperation interaction interface. The virtual reality controller integrates a motion state and an operation mode of the mobile robot, and the mixed reality server generates visual feedback according to visual information of a remote operation visual angle of the mobile robot; the robot end comprises a mobile robot, a collection device on the mobile robot and a robot server, and after the mobile robot receives a remote control instruction sent by the virtual reality controller to the robot end, posture and button information of the mobile robot are input into the robot server to drive the mobile robot to move. Therefore, the problems of low real-time performance and accuracy of remote control, poor operation efficiency and flexibility and the like caused by limitation of bandwidth and system time delay in related technologies are solved.

A computer implemented method for determining a 3D position of a surgical object relative to a region of interest in a patient's anatomy. The method includes receiving an X-ray image and a 3D data set. The X-ray image is a 2D projection image generated with an imaging direction and defining a 3D image coordinate system. The 2D projection image may depict a part of a patient's anatomy and a part of the surgical object. The received 3D data set describes the part of the patient's anatomy and a region of interest in a 3D data coordinate system. In order to determine a spatial relation between the 3D image coordinate system and the 3D data coordinate system, the part of the patient's anatomy is localized based on the X-ray image. Subsequently, the 3D position of the surgical object relative to the region of interest is determined in the 3D coordinate system.

The invention discloses a perspective display device and system for surgical navigation and mixed reality equipment. The device comprises an acquisition module for acquiring at least one two-dimensional X-film image comprising a target object and at least one marker, the marker and a three-dimensional virtual model of the target object; the pose determination module determines the poses of the markers and the target object in the X-ray imaging coordinate system based on the projection forms of the markers and the target object in the X-ray image, and the pose determination module comprises a spatial relationship determination unit which determines the spatial relationship between the markers and the target object based on the poses of the markers and the target object in the X-ray imaging coordinate system; the coordinate conversion module is used for converting the spatial relationship of the two in an X-ray imaging coordinate system into the spatial relationship in a mixed reality display coordinate system; and the perspective display module is used for determining the pose of the target object in the mixed reality display coordinate system based on the pose of the marker in the mixed reality display coordinate system and the spatial relationship, and displaying the three-dimensional virtual model of the target object and the operation scene in a virtual-real overlapping manner so as to realize operation navigation.

The invention discloses a vehicle control method and device, electronic equipment and a storage medium, and the method comprises the steps that mixed reality MR equipment displays a virtual space model corresponding to a vehicle, and the virtual space model is generated according to the internal space information of the vehicle; if the control information of the target equipment in the vehicle is obtained, displaying a virtual control panel of the target equipment pair in the virtual space model; if the control operation for the virtual control panel is obtained, control parameters are sent to the vehicle so that the vehicle can correspondingly control the target equipment based on the control parameters, and the control parameters are determined according to the control operation, so that the three-dimensional interaction space of the vehicle-mounted scene is expanded through the vehicle-mounted MR equipment; a new and more intuitive interaction mode is expanded for the user to control the vehicle, and the operation intuition and immersive interaction experience of the user are improved.

The present invention comprises an interface unit for receiving sensing information collected from a sensor, a tile database including tiles corresponding to at least one of an image and a shape of a part of a building, a memory for storing map information including virtual objects corresponding to buildings, and building profiles of the buildings corresponding to the virtual objects included in the map information, and a processor for performing control to: when a driving path of a vehicle is determined, detect building profiles corresponding to buildings around the vehicle on the basis of the driving path; on the basis of tile information for each area of the building profiles corresponding to virtual objects of the respective buildings around the vehicle, tile different areas of the respective virtual objects by using different tiles of the tile database; combine each tiled area; and display, on a display unit of the vehicle, MR information obtained by performing texturing on each virtual object.

The invention relates to the technical field of augmented reality, in particular to a 24-solar-term astronomical space-time presentation method and system based on mixed reality. The method comprises the following steps: firstly, reading augmented reality scene initialization data, configuring virtual scene parameters, and executing three-dimensional scene registration to generate a registration matrix; executing visual tracking by collecting an image stream, and generating a viewpoint transformation matrix; the system performs real-time sky semantic segmentation on an image stream to generate a binarized sky mask, and constructs an augmented reality rendering pipeline: synthesizing registration and viewpoint matrix calculation rendering operators, using the sky mask as an Alpha mask to perform pixel-level fusion of a virtual starry sky and a real sky, and combining sparse point clouds to perform depth occlusion synthesis, so as to obtain an augmented reality rendering pipeline. An atmospheric optical coloring model and a dynamic volume particle system are loaded for environmental atmosphere enhancement; and optical consistency fusion of a virtual image and a real environment is further realized through HDR luminosity alignment and tone mapping, and finally high-fidelity augmented reality superposition is realized.

The invention discloses a remote rail fault intelligent detection system based on big data driving, and relates to the technical field of rail traffic intelligent monitoring and state maintenance. The problems that a traditional detection method is high in manual dependence, slow in response and lack of prediction and self-repairing capacity are solved. According to the method, multi-source data are acquired through a fixed sensing network and a mobile inspection vehicle, a multi-scale digital twin is constructed and updated, and a probabilistic fault evolution graph is generated based on a random phase field model; collaborative judgment is carried out based on distributed edge computing nodes, and a maintenance scheme is generated by adopting a hybrid geometric distance retrieval immune memory bank; and finally, collaborative repair is executed through mixed reality and a micro-robot cluster, and continuous optimization of the system is realized based on federated learning and case-based reasoning. According to the system, integrated closed loop of track state sensing, fault prediction, intelligent decision making and self-repairing is achieved, and the detection foreseeability, the maintenance accuracy and the system self-adaptive capacity are improved.

The invention relates to the field of operation and maintenance management of Internet of Things equipment, in particular to a periodic maintenance task automatic scheduling system and method, and the method comprises the steps: firstly collecting vibration, current and environment signals, and constructing a three-dimensional digital twin map; obtaining a health index and a remaining available life in a preset window through time sequence alignment convolution and an ordinary differential equation recursive network, generating a risk coefficient through index mapping, and driving a reaction-diffusion equation to form a continuous degradation field; the scheduling optimization unit obtains an initial solution through symplectic gradient Hamiltonian-Monte Carlo sampling, and maintenance scheduling data meeting resource constraints are generated through bose sampling, topological bose evolution and dual gaming; the mixed reality terminal displays a risk hotspot and an operation instruction on site, retest data is recorded after maintenance is completed, and reinforcement learning updates a risk assessment and scheduling model on line according to an execution result to form a closed loop; according to the method, accurate quantification of risks, global optimization of scheduling and adaptive feedback of execution are realized, and the fault rate and the maintenance cost can be remarkably reduced.

The invention discloses a cross-platform MR scene collaborative editing method and system. The method comprises the steps that an access local area network generates and broadcasts a session ID; the head-mounted display end or the non-head-mounted display end scans a local network to discover and join the session; the head-mounted display end obtains space information and scene data bound from the platform and serializes the space information and the scene data into a unified standard format different from the platform; analyzing the data in the standard format to perform virtual scene reconstruction and spatial alignment; and creating a main state snapshot list, and performing maintenance synchronization on the edited state change by the host end or the server. The method has the advantages that platform barriers of various terminals such as a VR head-mounted display, a mobile tablet computer and a PC are broken through, and synchronous presentation and editing of the same mixed reality scene on heterogeneous equipment are achieved;

A method for sharing a display of a mixed reality experience is described. In one aspect, the method includes pairing a head-wearable device with a handheld device to establish a shared coordinate system, and displaying, based on a pose of the handheld device, a virtual object of a first mixed reality application operating at the head-wearable device, with a second mixed reality application operating at the handheld device using the shared coordinate system.

An automotive mixed-reality gaming system for a vehicle includes vehicle sensors that collect vehicle data associated with movements and a position of the vehicle along a roadway, detection sensors that collect environmental data associated with an external environment of the vehicle, and a processing unit including a processor and a memory. The processing unit receives data from the vehicle sensors and the detection sensors, converts the environmental data to a game-centric coordinate system stored in the memory, and generates an interactive gameplay application including a mixed-reality environment and a virtual avatar based on the game-centric coordinate system. Further, the processing unit transmits the application to mobile devices, receives user input from the mobile devices, and controls the activity of the avatar and settings of the gameplay based on the vehicle data and user input. The avatar traverses the mixed-reality environment at a rate proportional to the vehicle.

The present invention provides a route guidance device and a route guidance system. The route guidance device according to an embodiment of the present invention comprises: a communication part that communicates with a cloud server; an interface part that receives, from at least one sensor provided in a vehicle, a camera image including an image of a road on which the vehicle is traveling, and sensing information obtained by sensing a driving state of the vehicle; an MR module that renders MR information including at least one virtual object on the basis of the camera image, the sensing information, and map information received from the cloud server; and a processor that controls the interface part such that an MR view image including the MR information is displayed on a display part of the vehicle, wherein the processor, when a preset condition is satisfied, changes the MR view image to display a scene corresponding to a place at which the vehicle is to travel.

A see-through near eye display, which can be modulated in such a manner to synchronize with electronic enabled micro-lenses of a switchable micro-lens array, wherein when pixels of the near eye display are lit the micro-lenses of the micro-lens array are present; thus, a virtual image can be formed and seen by the eye of a user. When the see-through near eye display is not lit, the micro-lenses are not present, thus permitting a real image to be present.

The invention relates to the technical field of remote single-hole robot operation guidance and demonstration, and discloses a system and a method for remote single-hole robot operation guidance and demonstration, which realize efficient, stable and secure transmission of remote force feedback signals and control instructions through a low-delay secure communication link based on SRv6 + TSN and a national password SM9 encryption technology. The problem that transparency and real-time performance are difficult to consider in remote operation is effectively solved, experts can seamlessly perceive interaction force of local instruments and tissues and carry out accurate remote intervention, and through dynamic modeling of a cloud twinning service module and Transformer and graph neural network algorithms, the interaction force between the local instruments and the tissues can be accurately and remotely intervened. Real-time prediction of risks such as bleeding and perforation in the operation and accurate matching of tissue deformation are achieved, a visual and fused AR guidance interface is provided for experts in combination with a mixed reality guidance engine, and the accuracy of remote guidance and operation safety are remarkably improved.

A wearable augmented reality device enables rapid, personalized messaging through a body-anchored communication interface. The device captures image data via cameras, detects and tracks a body part of the wearer, and displays a user interface anchored to the detected body part. The interface comprises user interface elements representing social connections. Upon detecting selection of an element, the device activates a microphone to record an audio message for delivery to the corresponding social connection(s). The interface may be dynamically updated to reflect changes in relationship attributes, messaging activity, or connection status. User interface elements can be ordered based on social connection scores or communication recency, with additional graphical elements indicating device types or custom avatars. This system facilitates seamless integration of digital communication with the physical world, enhancing social media and instant messaging experiences in mixed reality environments.

Provided in the present invention are a virtual-robot and physical-robot collaborative safety control method and apparatus based on mixed reality and a digital twin, which method and apparatus specifically relate to the technical field of smart manufacturing and assembly involving virtual-robot and physical-robot collaboration. In the solution, the method comprises: acquiring first posture information of a physical robot, and acquiring second posture information of a virtual robot; on the basis of the first posture information and the second posture information, performing pose registration on the physical robot and the virtual robot, so as to obtain a registration result; on the basis of the registration result and a preset virtual-robot and physical-robot workspace in combination with deep reinforcement learning, performing path planning to determine a virtual-robot and physical-robot collaborative motion path; and on the basis of the virtual-robot and physical-robot collaborative motion path, executing a virtual-robot and physical-robot collaborative safety control action. The solution uses a mixed-reality device to extract virtual-robot and physical-robot posture information, implements virtual-robot and physical-robot pose registration by means of a virtual-real space mapping method, and designs a bidirectional virtual-robot and physical-robot collaborative safety interaction policy based on deep reinforcement learning, thereby effectively improving the safety and effectiveness of a virtual-robot and physical-robot interaction operation.

The invention provides an outdoor mixed reality illumination compensation method and system based on multi-source environment perception and real-time weather data, and the method comprises the steps: obtaining multi-source data: providing original data of'space-time reference + environment characteristics' for all subsequent illumination calculation, solving the problems of incomplete coverage and insufficient precision of a single data source (such as only a camera), and obtaining the multi-source data; the data is a data basis for real-time illumination compensation; main light source resolving: the'spatio-temporal data 'is converted into'virtual main light source parameters' through a physical algorithm, it is ensured that the direction and intensity of a virtual main light source (sun / moon) are completely consistent with those of the real world, and the problems of'shadow direction error 'and'illumination angle splitting' are solved; through GPS space-time positioning, network weather API data and local visual sensor fused illumination compensation, a real-time environment illumination shadow relationship matches a virtual world with a real world, a more immersive mixed reality experience is presented, and a split feeling is reduced.