161 results about "Artificial reality" patented technology

Systems and methods are provided for facilitating an interactive artificial-reality activity. A method includes, after a user of a head-wearable device has opted-in to using an artificial-reality application to facilitate connecting with other participating users, determining, based on user-specific suggestion criteria, that a suggested user is located in an approved common space with the user and has opted-in to use the artificial-reality application. The method includes causing the head-wearable device to present an user interface (UI) element for linking the suggested user and the user. Upon the user selecting the UI element for linking the suggested user with the user, the method includes automatically causing the head-wearable device to provide visual-guidance UI elements to navigate the user to an interactive-activity location where the user and the suggested user will perform an artificial-reality activity while also displaying information about the artificial-reality activity to be performed.

Example implementations are directed to recapturing or realigning a room or scene in an artificial reality (XR) environment as a three dimensional (3D) space. To initially capture a room, a user manually annotates, using an XR system, one or more walls around the XR system. The XR system then aligns or localizes itself relative to the room. Once a room is captured and stored, and the user in the future enters the room again, the XR system may not be able to align itself in the room due to, for example, lighting conditions. Rather than require the user to manually recapture every wall captured during the initial room capture, the XR system can select one wall for recapture and, after receiving an annotation or marking of the one wall from the user, use the selection to realign the XR system relative to the room.

A genealogy system includes a server with memory and processors storing code that instructs the processors to store genealogy data and user profiles, providing a research platform for users. A remote client device, equipped with an image sensor and display, communicates with the server. The client device displays the genealogy research platform, allowing users to select a genealogy item for an artificial reality experience. Upon user command, the client device presents continually updating artificial reality images of an environment, overlaying a digital representation of the selected genealogy item on the artificial reality experience. This system seamlessly integrates genealogical research with artificial reality technology for an immersive user experience.

A computer implemented method for facilitating system user interface (UI) interactions in an artificial reality (XR) environment is provided. The method includes rendering the system UI in the XR environment as a 3D virtual element. The method further includes tracking a position of a hand of a user and a pre-defined stable point on the user. The method further includes identifying, based on the tracking, that the hand has grasped a portion of the system UI and, in response, rotating the position of the system UI around the grasped portion of the system UI such that a line, between the stable point and the surface of the system UI, is moved, to be perpendicular or at a predefined angle from perpendicular to the surface of the system UI, as the user moves the system UI via the grasped portion.

Aspects of the present disclosure are directed to application multitasking in a shell with a three-dimensional environment. Implementations immerse a user in a three-dimensional environment via an artificial reality system, such as a display environment for a system shell. The system shell can execute applications (e.g., system shell applications, remoted applications, etc.). An executing application can correspond to a displayed virtual object (e.g., panel). The system shell can concurrently execute two, three, or more applications and the three-dimensional environment can concurrently display two, three, or more corresponding virtual objects that display contents for the executing applications. Implementations of a mode manager can manage a mode for the three-dimensional environment / system shell. Example modes include cooperative mode and exclusive mode. Implementations of cooperative mode permit concurrent display of multiple virtual objects from different applications while the exclusive mode permits display of virtual objects only from the executing application entering exclusive mode.

A computer implemented method for location-based interaction in an artificial reality (XR) environment comprises determining an indication of a location. A historical state of the location is retrieved in the XR environment. One or more inputs specifying a command related to the historical state of the location, are received on an XR device. The command indicates either to retrieve information of virtual entities that previously existed at the location or to undo a modification of one or more features at the location. One or more virtual entities are matched to the location in the XR environment for a given timeframe and an indication of the matched one or more virtual entities, is displayed.

Aspects of the present disclosure are directed to operating an artificial reality system in single-handed mode. Artificial reality systems receive user input via several channels, however conventional systems lack functionality that helps diverse users operate these systems. Some types of input, such as input that requires movement of two hands and / or two hand-held controllers, may be more challenging for some diverse individuals to provide or may not be possible in certain situations, e.g., where one controller is disabled. Implementations operate artificial reality systems in single-handed mode, such as by translating instances of single-handed input into two-handed input. For example, the translated two-handed input can cause application functionality at the artificial reality system that would otherwise pose a challenge for some diverse individual.

Aspects of the present disclosure are directed to triggering object transitions between near-field and far-field regions in an artificial reality environment. For example, the near-field region can be within arm's length for the user, while the far-field region can be greater than arm's length from the user. Objects can be displayed within these regions such that the user can interact with the objects in the artificial reality environment. A field manager can cause a displayed object to transition from the near-field region to the far-field region and / or from the far-field region to the near-field region. The field manager can compare sensed user movements to one or more trigger criteria (e.g., distance threshold(s), velocity threshold(s), etc.) to trigger transition of an object between field regions.

Artificial-reality devices may include left and right eyecups for respectively viewing left and right images. A flexible shroud membrane may be connected to the left and right eyecups. The artificial-reality devices may also include an interpupillary distance adjustment mechanism for adjusting a distance between the left and right eyecups and a force-offsetting mechanism configured to augment a force applied to adjust the distance between the left and right eyecups. Various other devices, systems, and methods are also disclosed.

An artificial reality system performs local user adaptation of machine learning models and global improvement of the machine learning models while ensuring data security and privacy. An artificial reality headset uses smart sensors and a system on chip (SoC). The smart sensor extracts features from sensor data, encrypts them and transmits the encrypted features to the SoC of the headset. The SoC provides the encrypted features to a server, for example, a server in a cloud platform. The server updates a base model with collected features and sends the updated base model parameters to headsets. The smart sensors of a headset receiving the updated model parameters, balance the model parameters with locally stored feature data, for example, user calibration features to perform user customization of the machine learning model.

A method of using time-of-flight sensors for gesture detection and content-rendering determinations in an artificial reality environment is provided. The method includes receiving data, from one or more time-of-flight sensors communicatively-coupled with a wrist-wearable device, about a physical surface, wherein the wrist-wearable device is communicatively-coupled with a head-wearable device that is configured to display a virtual object within an artificial-reality environment presented by the head-wearable device. The method also includes, in accordance with a determination, based on the data, that the physical surface has a curved surface portion, causing display of at least a portion of the virtual object at the curved surface portion, including updating the display of the virtual object in accordance with the curved surface portion.

Aspects of the present disclosure are directed to generating media element(s) using a multimodal scene graph. A scene manager can process visual information, such as video, images, and / or a recorded artificial relay scene, and generate a multimodal scene graph that comprises components and metadata generated via the processing. The scene manager can utilize the multimodal scene graph to generate social media elements, such as images, video, and / or artificial reality scenes. For example, a video of a user can be converted to a multimodal scene graph, which can be used to generate one or more images (e.g., memes, animated images, stickers, etc.), such as an image that represents the user via an avatar of the user. This generated media can be shared with other social platform users, and the stored multimodal scene graph can be accessed by the others to generate variations of the media.

The invention discloses automatic boundary creation and relocation. Aspects of the present disclosure relate to instant boundary creation for virtual reality (VR) experiences. An artificial reality (XR) system may scan and collect visual feature data of a user's real-world environment in the background while rendering an augmented reality (AR) environment or a mixed reality (MR) environment. When the XR system detects an intent to enter a VR mode (e.g., by launching a VR application), the XR system may generate a recommendation to a boundary of a real-world space, which may include a type of interaction mode (e.g., a movable mode or a stationary mode). Based on the user's response to the recommendation, the XR system may prompt the user to further scan the real-world space and / or manually adjust the boundaries while continuing to scan and collect visual feature data in the background. Alternatively or additionally, some embodiments may automatically relocate the real-world space based on the generated boundaries.

Systems and methods are provided for adjusting a representation of a user's position within an artificial-reality application using a hand gesture. One example method includes, while displaying, via a head-wearable device worn by a user, a representation of a user's position within an artificial-reality environment, in response to receiving an indication that a positional-control activation hand gesture has been performed, displaying a positional-control user interface (UI) overlaid on a portion of the artificial-reality environment, the positional-control UI including a positional-control UI element configured to perform a positional-control action. The example method further includes, while displaying the positional-control UI, in response to receiving an indication that the positional-control UI element has been selected, via a positional-control input hand gesture, causing a change in the representation of the user's position within the artificial-reality environment based on the positional-control action, and displaying a changed representation of the user's position within the artificial-reality environment.

Aspects of the present disclosure relate to automatic repositioning of virtual objects, in an augmented or mixed reality environment, to avoid occlusion of certain physical objects or views in the real-world environment. Conventionally, on a head-worn artificial reality system, head-leashed virtual objects simply follow where the head is pointed. Some implementations can detect regions in the field-of-view of the user that should not be occluded, such as faces of other people, media content, a particular activity being performed, and / or where the user's gaze has lingered, and reposition virtual objects to avoid these areas. Alternatively or additionally, some implementations can detect movement of the user in the real-world environment and trigger minimization, repositioning, and / or degradation of virtual objects to lessen obstructions in the user's viewpoint and conserve resources. After the user slows or stops moving, the virtual object can revert to its full form.

A headset, such as an artificial reality headset, includes a depth camera assembly that generates a three-dimensional model of a local area surrounding the headset. Additionally, the headset identifies objects in the local area through application of one or more trained models to images of the local area captured by imaging devices. The headset uses a bounding box determined for an identified object to map the identified object to the three-dimensional model of the local area. Based on the mapping, the headset may guide the user to the identified object or display content proximate to the identified object through a display element.

A computer implemented method for facilitating system user interface (UI) interactions in an artificial reality (XR) environment is provided. The method includes rendering the system UI in the XR environment as a 3D virtual element. The method further includes tracking a position of a hand of a user and a pre-defined stable point on the user. The method further includes identifying, based on the tracking, that the hand has grasped a portion of the system UI and, in response, rotating the position of the system UI around the grasped portion of the system UI such that a line, between the stable point and the surface of the system UI, is moved, to be perpendicular or at a predefined angle from perpendicular to the surface of the system UI, as the user moves the system UI via the grasped portion.

Aspects of the present disclosure are directed to controlling an immersive environment via an application programming interface (API). Some applications executing via artificial reality systems provide immersive content for display to the user. However, other types of artificial reality applications provide lighter weight content (e.g., two-dimensional content, three-dimensional content that is not immersive, etc.), such as a web browser, video player, social media application, communication application, and many others. These executing applications that provide content for portions of the artificial reality system's display often have limited control over the immersive elements of the artificial reality system, such as the immersive environment in which a two-dimensional or three-dimensional virtual object is displayed. Implementations of an immersive controller provide these applications an API for controlling these elements of the immersive environment via an API call.

Distributed real-time rendering is a cutting-edge solution to address the ever-increasing demands for realistic and immersive visual experiences, such as provided by augmented or artificial reality applications. Embodiments of the present specification provide a method and a system for distributed real-time rendering of a scene. The rendering is distributed among a root instance and a plurality of node instances. The root instance receives the scene to render, segments it into a plurality of objects, computes lighting parameters, assigns rendering tasks associated with objects to node instances, and communicates object metadata and the lighting parameters to the assigned node instance. The node instances render objects and create point clouds and u, v maps, which are then used by the root instance to render the scene for real-time display. The method and system can be used by end devices that include position sensors, processors, communication means and a display to provide immersive experiences.

Aspects of the present disclosure can trigger an action based on a motion detected by an artificial reality (XR) device, such as a head-mounted display (HMD). The XR device can display an XR experience to a user. While displaying the XR experience, the XR device can detect a physical interaction with the XR device using one of more sensors (e.g., sensors of an inertial measurement unit (IMU)). The physical interaction can generate a movement profile captured by the one or more sensors. The XR device can identify the physical interaction as a particular motion (e.g., one or more taps on the XR device) by applying a machine learning model to the movement profile. In response to identifying the particular motion, the XR device can trigger an action on the XR device (e.g., activating pass-through on the XR device).

A console and headset system locally trains machine learning models to perform customized online learning tasks. To customize the online learning models for specific users of the system without using outside resources, the system trains the models to compare a target frame to stored calibration frames, rather than directly inferring information about a target frame. During deployment, an embedding is generated for the target frame. A sample embedding that is closest to the target embedding is selected from a group of embeddings of calibration frames. The information about the selected embedding and target embedding and ground truths for the calibration frame are provided as inputs to one of the trained models. The model predicts a difference between the target frame and the calibration frame, which can be used to determine information about the target frame.

Aspects of the present disclosure relate to more accurate and quicker localization of an artificial reality (XR) system in a real-world space (e.g., a room). If a user enters a room and localization fails, the system can locate a corner that was designated in a previous localization. The corner could have been manually selected by the user or could have been automatically recommended by the XR system. In some implementations, the user or system can identify two adjacent corners in the room for further accuracy. Through later selection of the corner(s) for localization, the XR system can identify the saved room using depth sensors, with identification of corners being more reliable and detectable than other methods identifying walls.

Aspects of the present disclosure are directed to controlling a sending side of an augmented call based on a receiving user's gaze. Some implementations provide a hologram moderation system in which a receiving user's gaze can control how the system generates a representation of a sending user on a sending side. For example, some implementations can moderate the capture or generation of hologram data representing the sending user when the receiving user isn't focused on the hologram that results from the data. Such moderations can reduce power consumption, bandwidth, heat production, and / or processing power needed by the artificial reality system when the receiving user is not looking at the hologram of the sending user, such as when the sending user is in the receiving user's periphery or outside the receiving user's field-of-view.

To register an object for subsequent identification by an artificial reality headset, the headset receives an input from a user to enter a registration mode. The headset includes a plurality of imaging devices capturing images of a local area surrounding the headset. An eye tracking unit of the headset determines a gaze direction of the user, and the headset identifies an object in the local area where the user's gaze is directed and visually distinguishes the identified object to the user. In response to receiving a confirmation from the user to register the identified object, one or more images of the identified object are captured by the imaging devices and used to train an instance classifier to identify the identified object. The headset notifies the user when the identified object has been registered.

Aspects of the present disclosure provide a “framework of frameworks” to help developers build artificial reality (XR) applications, including two-dimensional and three-dimensional content, using disparate rendering frameworks. These rendering frameworks can output data to the XR environment, but cannot talk to each other. Thus, some implementations can provide an intermediary framework to coordinate communication and rendering of content between the various systems. The intermediary framework can provide input routing by detecting an event with respect to a piece of content, and routing input data only to the system associated with that piece of content. The intermediary framework can also allow a node within an augment associated with one system to receive notifications of events in another node within the augment associated with another system.

Aspects of the disclosure are directed to an artificial reality (XR) browser configured to trigger an immersive experience. Implementations display an element at a browser chrome of the XR browser when an immersive experience is loaded. For example, the XR browser can include an application programming interface (API) that supports configuration of a browser chrome element by components of a webpage. The API call can cause the display of the browser chrome element, change a display property for the browser chrome element, or configure the browser chrome element in any other suitable manner. Upon receiving input at the browser chrome element (configured by the API call), the XR browser can transition from displaying a two-dimensional panel view of a webpage supported by loaded web resources (e.g., hypertext transfer protocol (HTTP) pages, graphic images, etc.) to a three-dimensional environment supported by preloaded immersive resources (e.g., three-dimensional models, graphic images, etc.).

Various aspects of the subject technology relate to systems, methods, and machine-readable media for selective encryption in a shared artificial reality environment. Various aspects may include determining contextual information of the shared artificial reality environment. Aspects may also include encrypting communication in the environment into encrypted channels and non-encrypted channels based on the contextual information. Aspects may also include determining a correlation between the encrypted channels and the non-encrypted channels. Aspects may also include applying, based on the correlation, a partial encryption to the non-encrypted channels for obscuring a cryptographic code of the communication. Aspects may include determining a recombination of the encrypted channels and the non-encrypted channels based on clock skew.

Artificial reality (XR) experiences today typically only provide users representations of their upper body (e.g., as avatars). Although legs do not have a high range of movement or expression in XR, they are required to bring a sense of believability to digital humans represented in XR. However, tracking legs can be difficult because they are frequently not visible to XR device cameras. Aspects of the present disclosure provide a full body synthesis system that can generate plausible full body poses of users by leveraging generative machine learning, in real time, on an XR device. The full body synthesis system can be flexible to multiple numbers and types of inputs (e.g., positions / rotations / accelerations of joints, computer vision models, etc.), and can generalize users of any height, body scale, and body shape.

Artificial reality (XR) experiences today typically only provide users representations of their upper body (e.g., as avatars). Although legs do not have a high range of movement or expression in XR, they are required to bring a sense of believability to digital humans represented in XR. However, tracking legs can be difficult because they are frequently not visible to XR device cameras. Aspects of the present disclosure provide a full body synthesis system that can generate plausible full body poses of users by leveraging generative machine learning, in real time, on an XR device. The full body synthesis system can be flexible to multiple numbers and types of inputs (e.g., positions / rotations / accelerations of joints, computer vision models, etc.), and can generalize users of any height, body scale, and body shape.

Systems and methods are provided for facilitating an interactive artificial-reality activity. A method includes, after a user of a head-wearable device has opted-in to using an artificial-reality application to facilitate connecting with other participating users, determining, based on user-specific suggestion criteria, that a suggested user is located in an approved common space with the user and has opted-in to use the artificial-reality application. The method includes causing the head-wearable device to present an user interface (UI) element for linking the suggested user and the user. Upon the user selecting the UI element for linking the suggested user with the user, the method includes automatically causing the head-wearable device to provide visual-guidance UI elements to navigate the user to an interactive-activity location where the user and the suggested user will perform an artificial-reality activity while also displaying information about the artificial-reality activity to be performed.