113 results about "Artificial reality" patented technology

Example implementations are directed to recapturing or realigning a room or scene in an artificial reality (XR) environment as a three dimensional (3D) space. To initially capture a room, a user manually annotates, using an XR system, one or more walls around the XR system. The XR system then aligns or localizes itself relative to the room. Once a room is captured and stored, and the user in the future enters the room again, the XR system may not be able to align itself in the room due to, for example, lighting conditions. Rather than require the user to manually recapture every wall captured during the initial room capture, the XR system can select one wall for recapture and, after receiving an annotation or marking of the one wall from the user, use the selection to realign the XR system relative to the room.

Aspects of the present disclosure are directed to application multitasking in a shell with a three-dimensional environment. Implementations immerse a user in a three-dimensional environment via an artificial reality system, such as a display environment for a system shell. The system shell can execute applications (e.g., system shell applications, remoted applications, etc.). An executing application can correspond to a displayed virtual object (e.g., panel). The system shell can concurrently execute two, three, or more applications and the three-dimensional environment can concurrently display two, three, or more corresponding virtual objects that display contents for the executing applications. Implementations of a mode manager can manage a mode for the three-dimensional environment / system shell. Example modes include cooperative mode and exclusive mode. Implementations of cooperative mode permit concurrent display of multiple virtual objects from different applications while the exclusive mode permits display of virtual objects only from the executing application entering exclusive mode.

Aspects of the present disclosure are directed to operating an artificial reality system in single-handed mode. Artificial reality systems receive user input via several channels, however conventional systems lack functionality that helps diverse users operate these systems. Some types of input, such as input that requires movement of two hands and / or two hand-held controllers, may be more challenging for some diverse individuals to provide or may not be possible in certain situations, e.g., where one controller is disabled. Implementations operate artificial reality systems in single-handed mode, such as by translating instances of single-handed input into two-handed input. For example, the translated two-handed input can cause application functionality at the artificial reality system that would otherwise pose a challenge for some diverse individual.

Aspects of the present disclosure are directed to triggering object transitions between near-field and far-field regions in an artificial reality environment. For example, the near-field region can be within arm's length for the user, while the far-field region can be greater than arm's length from the user. Objects can be displayed within these regions such that the user can interact with the objects in the artificial reality environment. A field manager can cause a displayed object to transition from the near-field region to the far-field region and / or from the far-field region to the near-field region. The field manager can compare sensed user movements to one or more trigger criteria (e.g., distance threshold(s), velocity threshold(s), etc.) to trigger transition of an object between field regions.

An artificial reality system performs local user adaptation of machine learning models and global improvement of the machine learning models while ensuring data security and privacy. An artificial reality headset uses smart sensors and a system on chip (SoC). The smart sensor extracts features from sensor data, encrypts them and transmits the encrypted features to the SoC of the headset. The SoC provides the encrypted features to a server, for example, a server in a cloud platform. The server updates a base model with collected features and sends the updated base model parameters to headsets. The smart sensors of a headset receiving the updated model parameters, balance the model parameters with locally stored feature data, for example, user calibration features to perform user customization of the machine learning model.

A method of using time-of-flight sensors for gesture detection and content-rendering determinations in an artificial reality environment is provided. The method includes receiving data, from one or more time-of-flight sensors communicatively-coupled with a wrist-wearable device, about a physical surface, wherein the wrist-wearable device is communicatively-coupled with a head-wearable device that is configured to display a virtual object within an artificial-reality environment presented by the head-wearable device. The method also includes, in accordance with a determination, based on the data, that the physical surface has a curved surface portion, causing display of at least a portion of the virtual object at the curved surface portion, including updating the display of the virtual object in accordance with the curved surface portion.

Aspects of the present disclosure are directed to generating media element(s) using a multimodal scene graph. A scene manager can process visual information, such as video, images, and / or a recorded artificial relay scene, and generate a multimodal scene graph that comprises components and metadata generated via the processing. The scene manager can utilize the multimodal scene graph to generate social media elements, such as images, video, and / or artificial reality scenes. For example, a video of a user can be converted to a multimodal scene graph, which can be used to generate one or more images (e.g., memes, animated images, stickers, etc.), such as an image that represents the user via an avatar of the user. This generated media can be shared with other social platform users, and the stored multimodal scene graph can be accessed by the others to generate variations of the media.

The invention discloses automatic boundary creation and relocation. Aspects of the present disclosure relate to instant boundary creation for virtual reality (VR) experiences. An artificial reality (XR) system may scan and collect visual feature data of a user's real-world environment in the background while rendering an augmented reality (AR) environment or a mixed reality (MR) environment. When the XR system detects an intent to enter a VR mode (e.g., by launching a VR application), the XR system may generate a recommendation to a boundary of a real-world space, which may include a type of interaction mode (e.g., a movable mode or a stationary mode). Based on the user's response to the recommendation, the XR system may prompt the user to further scan the real-world space and / or manually adjust the boundaries while continuing to scan and collect visual feature data in the background. Alternatively or additionally, some embodiments may automatically relocate the real-world space based on the generated boundaries.

Systems and methods are provided for adjusting a representation of a user's position within an artificial-reality application using a hand gesture. One example method includes, while displaying, via a head-wearable device worn by a user, a representation of a user's position within an artificial-reality environment, in response to receiving an indication that a positional-control activation hand gesture has been performed, displaying a positional-control user interface (UI) overlaid on a portion of the artificial-reality environment, the positional-control UI including a positional-control UI element configured to perform a positional-control action. The example method further includes, while displaying the positional-control UI, in response to receiving an indication that the positional-control UI element has been selected, via a positional-control input hand gesture, causing a change in the representation of the user's position within the artificial-reality environment based on the positional-control action, and displaying a changed representation of the user's position within the artificial-reality environment.

A console and headset system locally trains machine learning models to perform customized online learning tasks. To customize the online learning models for specific users of the system without using outside resources, the system trains the models to compare a target frame to stored calibration frames, rather than directly inferring information about a target frame. During deployment, an embedding is generated for the target frame. A sample embedding that is closest to the target embedding is selected from a group of embeddings of calibration frames. The information about the selected embedding and target embedding and ground truths for the calibration frame are provided as inputs to one of the trained models. The model predicts a difference between the target frame and the calibration frame, which can be used to determine information about the target frame.

Aspects of the present disclosure relate to more accurate and quicker localization of an artificial reality (XR) system in a real-world space (e.g., a room). If a user enters a room and localization fails, the system can locate a corner that was designated in a previous localization. The corner could have been manually selected by the user or could have been automatically recommended by the XR system. In some implementations, the user or system can identify two adjacent corners in the room for further accuracy. Through later selection of the corner(s) for localization, the XR system can identify the saved room using depth sensors, with identification of corners being more reliable and detectable than other methods identifying walls.

Aspects of the present disclosure provide a “framework of frameworks” to help developers build artificial reality (XR) applications, including two-dimensional and three-dimensional content, using disparate rendering frameworks. These rendering frameworks can output data to the XR environment, but cannot talk to each other. Thus, some implementations can provide an intermediary framework to coordinate communication and rendering of content between the various systems. The intermediary framework can provide input routing by detecting an event with respect to a piece of content, and routing input data only to the system associated with that piece of content. The intermediary framework can also allow a node within an augment associated with one system to receive notifications of events in another node within the augment associated with another system.

Various aspects of the subject technology relate to systems, methods, and machine-readable media for selective encryption in a shared artificial reality environment. Various aspects may include determining contextual information of the shared artificial reality environment. Aspects may also include encrypting communication in the environment into encrypted channels and non-encrypted channels based on the contextual information. Aspects may also include determining a correlation between the encrypted channels and the non-encrypted channels. Aspects may also include applying, based on the correlation, a partial encryption to the non-encrypted channels for obscuring a cryptographic code of the communication. Aspects may include determining a recombination of the encrypted channels and the non-encrypted channels based on clock skew.

Systems and methods are provided for facilitating an interactive artificial-reality activity. A method includes, after a user of a head-wearable device has opted-in to using an artificial-reality application to facilitate connecting with other participating users, determining, based on user-specific suggestion criteria, that a suggested user is located in an approved common space with the user and has opted-in to use the artificial-reality application. The method includes causing the head-wearable device to present an user interface (UI) element for linking the suggested user and the user. Upon the user selecting the UI element for linking the suggested user with the user, the method includes automatically causing the head-wearable device to provide visual-guidance UI elements to navigate the user to an interactive-activity location where the user and the suggested user will perform an artificial-reality activity while also displaying information about the artificial-reality activity to be performed.

A method of interacting with an artificial-reality (AR) content at an AR headset that includes cameras and displays is described. The method includes, while the AR headset has a first position within a physical environment that includes an object, if a distance of the object is within a threshold collision distance from the AR headset, presenting a meshed representation of the object. The meshed representation is displayed at first respective locations on the displays such that the meshed representation is viewable within the artificial reality. After the AR headset moves to a second position with a different distance, and if the different distance is within the threshold collision distance from the AR headset, moving the meshed representation of the object to second respective locations within the artificial reality. The second respective locations correspond to the position of the object in the physical environment.

The present invention provides a DJ system for processing audio data, comprising an audio input unit for receiving audio data representing a piece of music, an audio processing unit for processing the audio data, an audio output unit for playing the audio data, and a gesture detecting apparatus for detecting a hand position, which is a position in space of a hand of a user of the system, and wherein the gesture detecting apparatus is configured to detect a turntable approach gesture or a transport control gesture.

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system can include a menu that can be activated and interacted with using one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system can cause a menu to be rendered. A menu sliding gesture (e.g., horizontal motion) of the hand can be used to cause a slidably engageable user interface (UI) element to move along a horizontal dimension of the menu while horizontal positioning of the UI menu is held constant. Motion of the hand orthogonal to the menu sliding gesture (e.g., non-horizontal motion) can cause the menu to be repositioned. The implementation of the artificial reality system does require use of both hands or use of other input devices in order to interact with the artificial reality system.

A head-mounted display including multiple displays tiled together is provided. A first display may include an active region and an inactive region. A second display may be aligned with the inactive region. The first display and the second display may have different resolutions and may present visual information with different display qualities based on their respective resolutions. For example, the first display may include a lower resolution than the second display. The inactive region may be centered in the first display, and the second display may be centered with respect to the first display. The head-mounted display may provide a continuous image(s) (or video(s)) with a high resolution on the second display, while providing the remainder of the image(s) on the low resolution first display. Based on the multiple displays being tiled, the multiple displays may present one or more images, or one or more videos, in a non-overlapping manner.

The embodiment of the invention relates to virtual space configuration based on gestures. A virtual space configuration system of an artificial reality system can detect user gestures and provide various corresponding customizations of the virtual space of the system. The virtual space configuration system may provide seating virtual space customizations when the user is in a seating position. In various implementations, these customizations may include: allowing adjustment of floor height; a mechanism to set a flag that can be revealed to the application to adjust the application for the seated user; customizing the display of virtual space boundaries while in the seating mode to reduce interference; providing an option to detect when the user leaves the seating mode and trigger a corresponding action; providing a delivery workspace region allowing a user to naturally interact with certain real-world objects without removing the virtual reality headset; or automatically determining a virtual space size for the seated user.

Aspects of the present disclosure relate to coordinating conversations with an "active talk bar" displayed on an XR device for an artificial reality (XR) environment, whose control bar for voice conversations continuously exists in multiple XR experiences. A user can create a voice call with other users of a plurality of XR devices, end the call, manage participants of the call, adjust call volume, and the like through the active call bar. Users may also view which XR experiences other users within the group are accessing and may invite them to join their XR experience. The active talk bar may continuously exist on the user interface as the user moves between the experience and the application. In addition, no matter where other users are located, the users can continue voice communication in different experiences, that is, the users can participate in the voice communication without being located in the same experience.

Aspects of the present disclosure are directed to gesture-based user interfaces (UIs) for artificial reality (XR) messaging applications. By supplementing or replacing “gaze to tap” user interfaces with “ultra low friction” (ULF) gestures, a user is not required to repeatedly remove his focus from the real world to gaze at menu options in order to select them. The ULF gestures can include, for example, a single pinch motion to tell a messaging application to start recording a voice message. Releasing the pinch stops the recording and allows for editing, while a snap (or tug right) can stop the recording and send the message immediately. A tug left can delete the message unsent. Adding these ULF gestures to the messaging application's UI allows the user to fully engage with the application while maintaining visual focus on the real world, thus encouraging the user to remain connected through the XR system.

Aspects of the present disclosure relate to hearing enhancement controls and modes for artificial reality (XR) systems. A hearing enhancement system can provide a wearer of an XR system, such as augmented reality (AR) glasses, heightened hearing for a conversation by accentuating a particular voice for the wearer. In some implementations, the system can 1) select a particular voice to accentuate, 2) apply filter(s) that eliminate sounds other than the selected voice, and 3) set the amount of amplification for the filtered voice based on a determined amount of residual noise in the filtered voice signal, such that the residual amount of noise is obscured from the wearer. In some implementations, the system can set the amount of amplification instead based on a determined amount of ambient noise in an audio signal, then filter the audio signal to a level that keeps the resulting residual noise below the ambient noise.

A room manager can generate mappings for a real-world room that support a shared XR environment. For example, the real-world room can include real-world objects and surfaces, such as a table(s), chair(s), wall(s), door(s), window(s), etc. The room manager can generate XR object definitions based on information received about the real-world room, object(s), and surface(s). For example, the room manager can implement a flow that guides a user equipped with an XR system to provide information for the XR object definitions, such as real-world surfaces that map to the XR object(s), borders (e.g., measured using a component of the XR system), such as borders on real-world surfaces, semantic information (e.g., number of seat assignments at an XR table, size of XR objects, etc.), and other suitable information. Implementations generate previews of the shared XR environment, such as a local preview and a remote preview.

Aspects of the present disclosure are directed to displaying color fields over time by an artificial reality system. A method includes causing a head-wearable device worn by a user to present a virtual object by successively displaying a plurality of color fields including presenting a first color field and a second color field at a first display position so that the virtual object is at a world-locked location as viewed by the user from a first perspective. The method includes, in accordance with a determination, based on user motion data, that misalignment criteria are satisfied causing the head-wearable device to present the virtual object by successively displaying the plurality of color fields including presenting the first and / or second color fields of the virtual object at a second display position so that the virtual object is at the world-locked location as viewed by the user from a second perspective.

In one embodiment, a method by a computing system comprising a color camera and two monochrome cameras respectively associated with two eyes of a user includes computing a point cloud corresponding to a visible environment based at least on two stereoscopic grayscale images respectively captured by the two monochrome cameras, generating a mesh corresponding to the visible environment based on the computed point cloud, and generating two stereoscopic colorized images to be respectively displayed to the two eyes of the user, where each of the two stereoscopic colorized images is generated using (1) the mesh, (2) luminance information from one of the two stereoscopic grayscale images that is associated with the eye to which the stereoscopic colorized image is to be displayed, and (3) color information from a color input image captured by the color camera.

A user enters natural language text into a mobile phone. The mobile phone detects features of the user. An artificial reality engine modifies features of the user based on an intent derived from the natural language.

Aspects of the present disclosure are directed to application multitasking in a shell with a three-dimensional environment. Implementations immerse a user in a three-dimensional environment via an artificial reality system, such as a display environment for a system shell. The system shell can execute applications (e.g., system shell applications, remoted applications, etc.). An executing application can correspond to a displayed virtual object (e.g., panel). The system shell can concurrently execute two, three, or more applications and the three-dimensional environment can concurrently display two, three, or more corresponding virtual objects that display contents for the executing applications. Implementations of a mode manager can manage a mode for the three-dimensional environment / system shell. Example modes include cooperative mode and exclusive mode. Implementations of cooperative mode permit concurrent display of multiple virtual objects from different applications while the exclusive mode permits display of virtual objects only from the executing application entering exclusive mode.

Aspects of the present disclosure are directed to creating and administering artificial reality collaborative working environments and providing interaction modes for them. An XR work system can provide and control such artificial reality collaborative working environments to enable, for example, A) links between real-world surfaces and XR surfaces; B) links between multiple real-world areas to XR areas with dedicated functionality; C) maintaining access, while inside the artificial reality working environment, to real-world work tools such as the user's computer screen and keyboard; D) various hand and controller modes for different interaction and collaboration modalities; E) use-based, multi-desk collaborative room configurations; and F) context-based auto population of users and content items into the artificial reality working environment.