67 results about "Artificial reality" patented technology

Example implementations are directed to recapturing or realigning a room or scene in an artificial reality (XR) environment as a three dimensional (3D) space. To initially capture a room, a user manually annotates, using an XR system, one or more walls around the XR system. The XR system then aligns or localizes itself relative to the room. Once a room is captured and stored, and the user in the future enters the room again, the XR system may not be able to align itself in the room due to, for example, lighting conditions. Rather than require the user to manually recapture every wall captured during the initial room capture, the XR system can select one wall for recapture and, after receiving an annotation or marking of the one wall from the user, use the selection to realign the XR system relative to the room.

Various aspects of the subject technology relate to systems, methods, and machine-readable media for selective encryption in a shared artificial reality environment. Various aspects may include determining contextual information of the shared artificial reality environment. Aspects may also include encrypting communication in the environment into encrypted channels and non-encrypted channels based on the contextual information. Aspects may also include determining a correlation between the encrypted channels and the non-encrypted channels. Aspects may also include applying, based on the correlation, a partial encryption to the non-encrypted channels for obscuring a cryptographic code of the communication. Aspects may include determining a recombination of the encrypted channels and the non-encrypted channels based on clock skew.

The present invention provides a DJ system for processing audio data, comprising an audio input unit for receiving audio data representing a piece of music, an audio processing unit for processing the audio data, an audio output unit for playing the audio data, and a gesture detecting apparatus for detecting a hand position, which is a position in space of a hand of a user of the system, and wherein the gesture detecting apparatus is configured to detect a turntable approach gesture or a transport control gesture.

The embodiment of the invention relates to virtual space configuration based on gestures. A virtual space configuration system of an artificial reality system can detect user gestures and provide various corresponding customizations of the virtual space of the system. The virtual space configuration system may provide seating virtual space customizations when the user is in a seating position. In various implementations, these customizations may include: allowing adjustment of floor height; a mechanism to set a flag that can be revealed to the application to adjust the application for the seated user; customizing the display of virtual space boundaries while in the seating mode to reduce interference; providing an option to detect when the user leaves the seating mode and trigger a corresponding action; providing a delivery workspace region allowing a user to naturally interact with certain real-world objects without removing the virtual reality headset; or automatically determining a virtual space size for the seated user.

Aspects of the present disclosure are directed to gesture-based user interfaces (UIs) for artificial reality (XR) messaging applications. By supplementing or replacing “gaze to tap” user interfaces with “ultra low friction” (ULF) gestures, a user is not required to repeatedly remove his focus from the real world to gaze at menu options in order to select them. The ULF gestures can include, for example, a single pinch motion to tell a messaging application to start recording a voice message. Releasing the pinch stops the recording and allows for editing, while a snap (or tug right) can stop the recording and send the message immediately. A tug left can delete the message unsent. Adding these ULF gestures to the messaging application's UI allows the user to fully engage with the application while maintaining visual focus on the real world, thus encouraging the user to remain connected through the XR system.

A room manager can generate mappings for a real-world room that support a shared XR environment. For example, the real-world room can include real-world objects and surfaces, such as a table(s), chair(s), wall(s), door(s), window(s), etc. The room manager can generate XR object definitions based on information received about the real-world room, object(s), and surface(s). For example, the room manager can implement a flow that guides a user equipped with an XR system to provide information for the XR object definitions, such as real-world surfaces that map to the XR object(s), borders (e.g., measured using a component of the XR system), such as borders on real-world surfaces, semantic information (e.g., number of seat assignments at an XR table, size of XR objects, etc.), and other suitable information. Implementations generate previews of the shared XR environment, such as a local preview and a remote preview.

Aspects of the present disclosure are directed to displaying color fields over time by an artificial reality system. A method includes causing a head-wearable device worn by a user to present a virtual object by successively displaying a plurality of color fields including presenting a first color field and a second color field at a first display position so that the virtual object is at a world-locked location as viewed by the user from a first perspective. The method includes, in accordance with a determination, based on user motion data, that misalignment criteria are satisfied causing the head-wearable device to present the virtual object by successively displaying the plurality of color fields including presenting the first and / or second color fields of the virtual object at a second display position so that the virtual object is at the world-locked location as viewed by the user from a second perspective.

In one embodiment, a method by a computing system comprising a color camera and two monochrome cameras respectively associated with two eyes of a user includes computing a point cloud corresponding to a visible environment based at least on two stereoscopic grayscale images respectively captured by the two monochrome cameras, generating a mesh corresponding to the visible environment based on the computed point cloud, and generating two stereoscopic colorized images to be respectively displayed to the two eyes of the user, where each of the two stereoscopic colorized images is generated using (1) the mesh, (2) luminance information from one of the two stereoscopic grayscale images that is associated with the eye to which the stereoscopic colorized image is to be displayed, and (3) color information from a color input image captured by the color camera.

A user enters natural language text into a mobile phone. The mobile phone detects features of the user. An artificial reality engine modifies features of the user based on an intent derived from the natural language.

Aspects of the present disclosure are directed to application multitasking in a shell with a three-dimensional environment. Implementations immerse a user in a three-dimensional environment via an artificial reality system, such as a display environment for a system shell. The system shell can execute applications (e.g., system shell applications, remoted applications, etc.). An executing application can correspond to a displayed virtual object (e.g., panel). The system shell can concurrently execute two, three, or more applications and the three-dimensional environment can concurrently display two, three, or more corresponding virtual objects that display contents for the executing applications. Implementations of a mode manager can manage a mode for the three-dimensional environment / system shell. Example modes include cooperative mode and exclusive mode. Implementations of cooperative mode permit concurrent display of multiple virtual objects from different applications while the exclusive mode permits display of virtual objects only from the executing application entering exclusive mode.

Aspects of the present disclosure relate to providing assisted scene capture for an artificial reality (XR) environment. Some embodiments may obtain an XR space model corresponding to a real-world space, including walls, ceilings, and floors, using an XR system. Some embodiments may then scan the real-world space and display a grid overlying a view of the real-world space corresponding to the scanned area. Some embodiments may perform post-processing on the displayed mesh, which may include: A) folding a change in the mesh corresponding to a wall, ceiling, or floor onto the XR spatial model; the method comprises the steps of: A) selecting an XR spatial model, B) cutting a mesh to the XR spatial model for an open doorway, window or the like, C) simplifying the mesh corresponding to a wall, ceiling or floor, D) determining that the mesh is intact, or any combination thereof, by identifying that a threshold percentage of the XR spatial model is covered by the mesh.

Aspects of the present disclosure are directed to detecting and mitigating a degradation state at an artificial reality system. An artificial reality (XR) system can comprise several different input channels. The performance of one or more of these input channels, or channel fidelity, can degrade over time, such as in the presence of different operating conditions. Implementations can perform mitigation workflows to improve the channel fidelities of various input channels. When channel fidelity is improved, the input channel may be maintained as an active channel for receiving one or more types of user input. When channel fidelity is not improved, one or more other input channel(s) can be selected as the active channel(s) (e.g., replacement input channels) for receiving one or more types of user input.

A virtual space configuration system of a human reality system can detect user posture and provide various corresponding customizations of the virtual space of the system. When a user is in a seated posture, the virtual space configuration system can provide seated virtual space customizations. In various implementations, these customizations can include allowing adjustment of floor height, setting a flag that can be exposed to applications to adjust mechanisms of the applications for seated users, customizing display of virtual space boundaries while in seated mode to reduce interference, providing options to detect when a user leaves seated mode and trigger corresponding actions, providing a pass-through workspace region allowing a user to naturally interact with certain real-world objects without removing a virtual reality headset, or automatically determining virtual space dimensions for seated users.

Aspects of the present disclosure can modify a user interface of an artificial reality system based on an ascertained intent of the user. For example, some implementations can "lock" a user's virtual hand to a virtual object displayed on the user interface when a user intends to interact with it with their physical hand, instead of tracking the physical hand with the virtual hand. While the virtual hand is "locked" to the virtual object, some implementations can further A) render another representation of the physical hand, separate from the virtual hand, corresponding to the actual location of the physical hand, B) render a visual affordance on the virtual hand based on movement of the physical hand, and / or C) restrict the amount of wrist rotation that is shown by the virtual hand when the physical hand interacts with the virtual object.

Visual vergence determination is disclosed. In one embodiment, an artificial reality system determines that a performance metric of an eye tracking system is below a first performance threshold. The eye tracking system is associated with a head-mounted display worn by a user. The artificial reality system receives a first input associated with a body of the user and, based on the received first input, determines an area that the user is looking at within a field of view of the head-mounted display. The system determines a visual vergence distance of the user based on at least the first input associated with the body of the user, the area that the user is looking at, and a location of one or more objects in a scene displayed by the head-mounted display. The system adjusts one or more configurations of the head-mounted display based on the determined visual vergence distance of the user.

A packaging charging station is disclosed. A packaging charging station can include an outer portion, an inner portion, a receiving portion, and a connecting portion. The inner portion can include a mold corresponding to an artificial reality device. The receiving portion can be included in the mold at a location corresponding to a charging port of the artificial reality device. The connecting portion can be included in the outer portion, and the connecting portion can be configured to connect to an external power source to charge the artificial reality device when the artificial reality device is placed in the inner portion.

Distributed real-time rendering is a cutting-edge solution to address the ever-increasing demands for realistic and immersive visual experiences, such as provided by augmented or artificial reality applications. Embodiments of the present specification provide a method and a system for distributed real-time rendering of a scene. The rendering is distributed among a root instance and a plurality of node instances. The root instance receives the scene to render, segments it into a plurality of objects, computes lighting parameters, assigns rendering tasks associated with objects to node instances, and communicates object metadata and the lighting parameters to the assigned node instance. The node instances render objects and create point clouds and u, v maps, which are then used by the root instance to render the scene for real-time display. The method and system can be used by end devices that include position sensors, processors, communication means and a display to provide immersive experiences.

The present disclosure provides world-controlled augments and application-controlled augments. World-controlled augments can be controlled directly by a shell in the artificial reality environment. To allow even inexperienced users to develop world-controlled augments, a world-controlled builder system is provided. Application-controlled augments may be resource intensive (e.g., using eye-tracking, social-media tie-ins, etc.), may support complicated interactions among themselves, may require or have extensive use of inputs and permissioned resources, and are controlled by their hosting application. When a running application is halted, the application closes its application-controlled augments but can choose to have the XR system run a “place-holder” world-controlled augment for as long as the application is not running. The place-holder world-controlled augment preserves the appearance of the application in the artificial reality environment but uses few system resources. Upon a user selecting a place-holder world-controlled augment, the XR system can run the associated application and close the place-holder world-controlled augment.

A system generates a plurality of spatial points based on depth measurements of physical objects. The system determines, based on the plurality of spatial points, an occupancy score for each voxel within a plurality of voxels. The system identifies, based on a gaze of the user, a first set of occupied voxels that are in a field of view of the user and a second set of occupied voxels that are outside the field of view of the user. The system updates the occupancy scores of the first set of occupied voxels by temporally decaying one or more of the plurality of spatial points within the first set of occupied voxels. The system maintains the occupancy scores of the second set of occupied voxels. The system detects intrusions in a predefined subspace within a physical space based on the updated occupancy scores of the first set of occupied voxels.

A room manager can generate mappings for a real-world room that support a shared XR environment. For example, the real-world room can include real-world objects and surfaces, such as a table(s), chair(s), wall(s), door(s), window(s), etc. The room manager can generate XR object definitions based on information received about the real-world room, object(s), and surface(s). For example, the room manager can implement a flow that guides a user equipped with an XR system to provide information for the XR object definitions, such as real-world surfaces that map to the XR object(s), borders (e.g., measured using a component of the XR system), such as borders on real-world surfaces, semantic information (e.g., number of seat assignments at an XR table, size of XR objects, etc.), and other suitable information. Implementations generate previews of the shared XR environment, such as a local preview and a remote preview.

A virtual space configuration system of an artificial reality system can detect user posture and provide various corresponding customizations of the system's virtual space. The virtual space configuration system can provide sitting virtual space customizations when the user is in a sitting posture. In various implementations, these customizations may include allowing adjustment of floor height; setting flags that can be surfaced for applications to adjust the dynamics of applications for seated users; customizing the display of virtual space boundaries to be less obtrusive when in sitting mode; providing options to detect when the user leaves sitting mode and trigger corresponding actions; providing a pass-through workspace area that allows the user to naturally interact with certain real-world objects without needing to remove the virtual reality headset; or automatically determining virtual space dimensions for seated users.

Implementations generate personalized property recommendations based on experiential data obtained via artificial reality systems and machine learning predictions that correlate user preferences to these properties. A user can be one or more persons interested in purchasing a property, such as a new home. A property buying workflow often includes defining a set of preferences for a desired property and comparing properties for sale with these defined preferences. However, in a conventional buying workflow, defining preferences and doing the comparison is left up to the user and / or the user's real estate agent. Users can often feel overwhelmed by the information, in particular users that do not have previous buying experience. Implementations can integrate machine learning and artificial reality systems into the property buying experience to mitigate the information overload problem and provide meaningful recommendations to the user, such as a ranking of properties that comply with user's preferences.

Aspects of the present disclosure relate to two-dimensional (2D) user interface (UI) content overlay for artificial reality (XR) environments. In some implementations, when a user launches an immersive XR application that assists the user in navigating and controlling an immersive destination, the content overlay may be a default home screen and complete related actions and frequent actions involving the XR application. In some implementations, content overlay may serve dynamic content based on the user's context to assist users, such as: A) lookup and interaction with other users, B) understanding of related actions that may be taken, C) fast travel to a destination, D) viewing of related content, etc. Content overlay can provide these functional 2D cards in an XR environment, which can be specific to an executing XR application, or can be a system level that provides content for multiple XR applications.

The embodiment of the invention relates to virtual space configuration based on gestures. A virtual space configuration system of an artificial reality system can detect user gestures and provide various corresponding customizations of the virtual space of the system. The virtual space configuration system may provide seating virtual space customizations when the user is in a seating position. In various implementations, these customizations may include: allowing adjustment of floor height; a mechanism to set a flag that can be revealed to the application to adjust the application for the seated user; customizing the display of virtual space boundaries while in the seating mode to reduce interference; providing an option to detect when the user leaves the seating mode and trigger a corresponding action; providing a delivery workspace region allowing a user to naturally interact with certain real-world objects without removing the virtual reality headset; or automatically determining a virtual space size for the seated user.

Aspects of the present disclosure are directed to an artificial reality (XR) application system controlling applications in an artificial reality environment. In various cases, these controls include automatically suggesting XR applications by determining an XR context and identifying applications that match the XR context. These applications can be suggested to a user, who can authorize their execution, setting permissions for the application. In some cases, applications can be divided into components which can be progressively downloaded. By providing application suggestions relevant to the current context and progressively downloading application components, applications can appear ambient, rather than relying on users to constantly download, install, or activate applications. Permissions for applications may be revoked permanently or for certain situations—either through user permissions selections or automatically in response to determined user intents. When multiple applications are simultaneously authorized to execute, the XR application system can employ a ranking system to prevent overcrowding.

A user enters natural language text into a mobile phone. The mobile phone detects features of the user. An artificial reality engine modifies features of the user based on an intent derived from the natural language.

Aspects of the present disclosure are directed to providing assisted scene capture for an artificial reality (XR) environment. Some implementations can obtain an XR space model corresponding to a real-world space, including walls, the ceiling, and the floor, using an XR system. Some implementations can then scan the real-world space and display a mesh corresponding to the scanned area overlaid onto a view of the real-world space. Some implementations can perform post-processing on the displayed mesh, which can include A) collapsing variations in the mesh corresponding to the walls, ceiling, or floor onto the XR space model; B) clipping the mesh to the XR space model for open doorways, windows, etc.; C) simplifying the mesh corresponding to the walls, ceiling, or floor; D) determining that the mesh is complete by identifying that a threshold percentage of the XR space model is covered by the mesh; or any combination thereof.

A system suppresses howl in a device including microphones and speakers, for example, an artificial reality headset. A speaker of the device presents audio content. The audio content presented by the speaker is received by a microphone of the device thereby creating a howl in certain situations. The system detects the presence of the howl in a region of the audio content using an adaptive notch filter. The system suppresses the howl by reducing gain of one or more frequencies of the audio content. The system may detect presence of the howl by monitoring flatness of the signal. The system may detect presence of the howl based on tonality detection based on linear prediction.

Aspects of the present disclosure are for an interaction mode system that provides multiple interaction modes in an artificial reality environment with automatic, context-specific transitions between interaction modes. An interaction mode can specify how the interaction mode system determines direction indications and movement within an artificial reality environment and interactions for making selections or performing other actions. In some implementations, the interaction mode system can control at least four interaction modes including a no-hands, 3DoF mode, a no-hands, 6DoF mode, a gaze and gesture mode, and a ray casting mode. The interaction mode system can employ a mapping of interaction mode context factors (e.g., which components are enabled, mode settings, lighting or other environment conditions, current body positions, etc.) to interaction modes to control transitioning between particular interaction modes. The interaction mode system can also provide affordances for signaling a current interaction mode and interaction mode transitions.