Spatial behavior in a three-dimensional environment

Abstract

A system is disclosed for determining an input's position in a three-dimensional environment. In extended reality applications, a visual indicator corresponding to an input can disappear or jump erratically when traversing the void space between virtual objects, degrading the performance of the human-computer interface. The disclosed system addresses the technical problem by performing a geometric analysis to create a predictable trajectory for the input. When a user's input is in a void, the system projects nearby virtual objects onto a two-dimensional plane perpendicular to the input's direction to identify a set of nearest neighbors. The input's depth is then calculated as a combination of the depths of the neighbors. Separately, a corresponding indicator is selectively visualized based on the input's proximity to a selected virtual object in a second plane corresponding to the user's viewpoint, to have the indicator appear at a perceptually consistent distance and reduce visual clutter.

Description

Atty Docket No. 0120-1111WO1SPATIAL BEHAVIOR IN A THREE-DIMENSIONAL ENVIRONMENTCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application is a continuation of and claims priority to U.S. Provisional Application No. 63 / 736,299, filed December 19, 2024, the entire contents of which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Computing systems have evolved from text-based interfaces to sophisticated graphical user interfaces that allow users to interact with digital content visually. More recently, graphical user interfaces have expanded into three-dimensional (3D) environments, such as those found in extended reality (XR), mixed reality (MR), augmented reality (AR), and virtual reality (VR) systems. In immersive environments, users can view and manipulate virtual objects that are rendered within a 3D space. Such interaction commonly involves controlling a positional input, often rendered as an indicator or cursor, that is mapped to user signals like hand gestures, head orientation, or eye gaze.SUMMARY

[0003] A system is disclosed for determining a position for an input within a 3D virtual space. In such an environment, a user interacts with virtual objects through actions, such as an eye gaze or hand gesture, which a system detects to generate a directional signal. The system interprets the directional signal to define a computational input having an origin and a direction. When the line defined by the input's origin and direction traverses the empty space between virtual objects, a conventional system may fail to determine a depth for the input, causing a corresponding visual indicator to disappear or jump unpredictably. The disclosed system addresses the challenge by calculating a stable and predictable position for the input, even in the absence of a direct intersection with an object.

[0004] When an input is not intersecting a virtual object, the system determines a depth for the input by performing a geometric analysis of the input’s spatial relationship to nearby virtual objects. In an example implementation, a system computationally constructs a two-dimensional (2D) plane and determines a location for the input within that plane. Based on the input’s location,Atty Docket No. 0120-1111WO1 the system identifies at least one proximate virtual object and determines a depth for the input based on a depth of the identified at least one virtual object. The process creates a smooth and predictable trajectory for the input as the input moves through empty space.

[0005] In some implementations, the geometric analysis involves identifying multiple virtual objects and calculating the input’s depth through a combination of the depths of a subset of those objects. To perform such a calculation, a system may construct the 2D plane to be perpendicular to the direction of the input and project the 3D coordinates of the input and the virtual objects onto the plane. A geometric structure, such as a Voronoi diagram, can be generated on the plane to partition the plane and identify a subset of "nearest neighbor" virtual objects that surround the input's location. The depth of the input is then calculated as a weighted average of the depths of the virtual objects in the subset.

[0006] In addition to determining the input’s position, some implementations control when a visual indicator for the input is rendered. In an example, a system may select a single virtual object from a group of nearby objects based on calculated weights from the depth-determination process. A distance between the selected virtual object and the input is then determined in a second plane that corresponds to a user’s viewpoint. The system then renders the input as an indicator in response to the determined distance meeting a predefined threshold. The process allows the indicator to appear at a perceptually consistent distance from objects while minimizing visual clutter.

[0007] It is appreciated that the described systems and methods can include any combination of the aspects and features described herein and are not limited to the specific combinations set forth in the disclosure. Moreover, the details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following detailed description that sets forth aspects of the subject matter, along with the accompanying drawings of which:

[0009] FIG. 1 is a conceptual diagram illustrating an example extended reality environment where a system for determining an input's position may be implemented;Atty Docket No. 0120-1111WO1

[0010] FIG. 2A is a conceptual diagram illustrating a gaze-based input interacting with a virtual volume in an example implementation;

[0011] FIG. 2B is a conceptual diagram illustrating a hand-based input interacting with a virtual volume in an example implementation;

[0012] FIG. 3 is a conceptual diagram illustrating a two-dimensional plane used for determining an input’s depth via interpolation based on neighboring virtual objects in an example implementation;

[0013] FIG. 4 is a conceptual diagram illustrating a visibility area for controlling visualization of an input based on proximity to a virtual object in an example implementation;

[0014] FIG. 5 is a flowchart illustrating a representative process for determining an input’ s depth in a virtual environment;

[0015] FIG. 6 is a flowchart illustrating a representative process for controlling visualization of an input in a virtual environment; and

[0016] FIG. 7 is a block diagram illustrating an example computing system that may be employed to provide an implementation of the disclosed methods.

[0017] Like reference numbers may refer to like elements.DETAILED DESCRIPTION

[0018] In an extended reality (XR) environment, a computer system renders virtual objects, such as application windows or 3D content, that are sparsely arranged in a 3D space. User interaction is facilitated through a computational input, rendered as a visual indicator, which is generated from sensor data corresponding to user actions. A specific technical problem arises when the computational input traverses the void space between virtual objects. In such a scenario, conventional systems lack surface intersection data, a failure that prevents the system from determining a valid depth coordinate for the input. The data deficit leads to an undefined state in the rendering pipeline, causing the visual indicator to be rendered at an erroneous position or not rendered at all. The resulting output is unpredictable and computationally inefficient, as the system cannot maintain a stable, continuous trajectory for the input, thereby degrading the performance of the human-computer interface.

[0019] The interaction challenge in a sparse 3D environment presents a core technical problem rooted in specific failures of data processing and rendering pipelines in conventional XR computerAtty Docket No. 0120-1111WO1 systems. When an input raycast fails to intersect a virtual object, the system suffers from a data deficit, a condition where no geometric intersection data is available to determine the input’s depth. The lack of data results in an undefined state for a system variable, which causes the rendering pipeline to produce an erroneous and unpredictable output, such as a jarring visual jump or disappearance of the input indicator. Conventional workarounds introduce further technical flaws. One common approach, programmatically enlarging the invisible collision volumes (colliders) around objects, can be computationally inefficient. The approach increases the number of potential collision-check calculations the processor must perform on every frame, consuming processor cycles and battery power. Furthermore, the approach creates overlapping target zones that lead to computational ambiguity and logical failures, where the computer incorrectly interprets user intent by selecting the wrong object. Such methods are purely reactive and fail because the underlying techniques lack a mechanism to generate the necessary data to maintain a stable system state.

[0020] In some implementations, the disclosed methods provide a specific technical solution that improves the functioning of an XR computer system by overcoming the data deficit. The solution is not merely an abstract idea of interpolation, but an unconventional data generation technique that alters the computer’s operational state and its user interface rendering system. The ordered combination of steps is unconventional and not routine in the art because, instead of reacting to existing data from an intersection point, which is the conventional approach, the system proactively generates a new, technically significant data point, the interpolated depth, by performing a specific geometric analysis on a set of non-intersected, proximate data points. The process comprises: identifying an input in a virtual space, the input defined by an origin and a direction; determining a location of the input in a plane; identifying at least one virtual object based on at least one criterion associated with the location of the input; and determining a depth of the input based on a depth of the at least one virtual object. A specific, ordered process may involve: (1) for a given input traversing a void, identifying a set of nearby virtual objects; (2) computationally projecting the 3D coordinates of the objects and the input onto a 2D plane that is perpendicular to the input’s direction; (3) generating a Voronoi diagram on the plane to identify a specific subset of “nearest neighbor” objects that geometrically surround the input’s projected location; and (4) calculating the input’s precise depth through a combination of the depths of the nearest neighbors, for example, by determining a weighted average, specific, ordered process involves: (1) for a given input traversing a void, identifying a set of nearby virtual objects; (2)Atty Docket No. 0120-1111WO1 computationally projecting the 3D coordinates of the objects and the input onto a 2D plane that is perpendicular to the input’s direction; (3) generating a Voronoi diagram on the plane to identify a specific subset of “nearest neighbor” objects that geometrically surround the input’s projected location; and (4) calculating the input’s precise depth through a combination of the depths of the nearest neighbors, for example, by determining a weighted average. The process represents a specific improvement to the computer’s functionality, not a well-understood, routine, or conventional activity, because the process enables the computer to perform a new function: maintaining a stable and predictable state variable (the input’s 3D position) even in the absence of direct input data, a function that conventional XR systems cannot perform.

[0021] A technical advantage of the unconventional process is the creation of a smooth, deterministic trajectory for an input through empty space, which resolves the undefined state that can affect conventional systems. The technique represents a specific alteration to a computer’s functionality by transforming the computer’s operational paradigm for processing user signals from a purely reactive model, which depends on direct geometric intersection, to a predictive, contextual model that can generate valid positional data in a void. Furthermore, the system repurposes data from the depth interpolation process to solve a separate visualization problem. Specifically, the system selects a single virtual object based on the calculated interpolation weights and uses a 2D distance to that object to control the indicator’s visibility. Repurposing the weight data can reduce the computational load, as the system avoids a redundant spatial analysis, thereby conserving processor cycles and reducing power consumption. The approach enables a computer to render a visible indicator at a stable, appropriate depth even when no object is being directly intersected, a function the computer could not otherwise perform. The technique alters the functioning of an XR system by changing how the system processes user signals and renders interface elements, transforming the system from one that fails in sparse environments to one that provides a stable and predictable means of navigation.

[0022] As used herein, an input refers to a computational entity in a 3D virtual space that is defined by an origin and a direction. For clarity and consistency with terminology common in the art, an input may also be referred to herein as an input vector or a raycast. The calculated position, location, and depth of the input determines the rendering of a corresponding visual indicator. An input may be identified based on a user signal, which is a digital signal generated by one or more sensors corresponding to a user action, such as a detected eye gaze, a detected head gaze, or aAtty Docket No. 0120-1111WO1 detected hand position. An indicator may refer to a visual representation, such as a cursor, that is rendered at the calculated position of the input. A virtual object may refer to any computergenerated element rendered in the virtual space with which a user can potentially interact. A criterion may refer to a set of one or more rules used to select a virtual object.

[0023] As used herein, a plane may refer to a 2D geometric surface constructed within a virtual space for computational purposes. Depth may refer to a calculated scalar value representing a distance from an origin, measured along an axis parallel to an input’s direction. A subset of virtual objects may refer to a group of virtual objects identified based on a specific property, such as a proximity to an input's location in a plane. Proximity may refer to a condition satisfied when a calculated distance between an input’s location and a virtual object in a plane is less than a predetermined value. A Voronoi diagram may refer to a tessellation of a plane that partitions the plane into polygonal regions, or cells, based on the proximity to a set of points. A combination may refer to a calculation used to determine an input’s depth from the depths of a subset of virtual objects. A weighted average is one example of a combination that may be used, where each virtual object's contribution is scaled by a calculated weight.

[0024] As used herein, a selected virtual object may refer to a single virtual object selected from a subset of virtual objects based on a set of determined weights. A second plane may refer to a 2D plane that corresponds to a user's viewpoint. An interaction may refer to a user action with a selected virtual object, such as a drag operation where a user selects the object and moves the input while maintaining the selection. A threshold may refer to a predefined value, such as a distance in screen space, used as a condition for rendering an indicator.

[0025] As used herein, an indicator is a user-controlled graphical element, such as a cursor, that represents a point of interaction in a 3D environment, enabling the selection and manipulation of virtual objects. The indicator's 3D coordinates are computed based on a user's input and the input's spatial relationship to nearby virtual objects. Depth may refer to a scalar value representing the position of a point in a 3D coordinate system along a specific axis, often corresponding to a distance from the input's origin. When an input is not intersecting any object, the depth is a value calculated via a combination of the depths of neighboring objects.

[0026] FIG. 1 is a third-person view of an example physical space 100, in which a user experiences an XR environment 102 through a display 10 of a head-mounted display (HMD) 11. The XR environment 102 can be generated by an XR application 120 and displayed to the userAtty Docket No. 0120-1111WO1 through the HMD 1 1 or another device. The XR environment 102 includes one or more virtual objects (VO) that can be selected by the user via an input, which may be rendered as an indicator X. In the example, there are three virtual objects, VO1-VO3, each having a unique 3D location in the XR environment 102. For example, VO1 could represent a video player application window, VO2 could be a social media feed, and VO3 could be a 3D model that the user is manipulating, with each object arranged at a different depth to create a spatial user interface. Although the illustrated implementation uses an HMD 11, other types of devices can be used, such as a mobile phone or laptop computer. In some implementations, a user could hold a device and view the XR environment through a display of the device.

[0027] As shown in FIG. 1, the user controls a position of an input, rendered as indicator X. The control modality is a variable, as the origin point of the input (106 or 106’) changes the geometric relationship between the input and the virtual objects. In a gaze-based modality, the input's origin 106 is located in the user’s head region and is therefore relatively stable. In contrast, for a handbased modality, the origin 106’ is dynamic, corresponding to the detected location of the user’s hand. The dynamic origin creates different parallax effects and interaction challenges compared to the stable origin of a gaze-based input. The system can be configured to handle the different modalities. For a gaze-based signal, the XR application 120 may define a 3D input 104 with a direction that corresponds to a detected head or eye gaze and an origin 106 in the user’s head region. For a hand-based signal, the XR application 120 may define a 3D input 104’ with a direction that corresponds to a detected hand position and an origin 106’ at the detected location of the user’s hand.

[0028] In some examples, an indicator corresponding to the input may not be visualized at all times. For example, the indicator may be rendered only when the input 104 is intersecting a virtual object. When the input is not intersecting a target, the indicator is not visualized by the XR application 120. The intermittent visualization of the indicator can, for example, minimize visual clutter in the XR environment 102. Not rendering the indicator, however, means the system fails to provide a continuous positional reference for the input, which can lead to inefficient interaction. For example, in the implementation shown in FIG. 1, there is void space (of the XR environment) between virtual objects VO1, VO2, and VO3. When the input is located in a void space, the system does not render an indicator, and a technical problem arises as there is no visual data representing the input's current location.Atty Docket No. 0120-1111WO1

[0029] A non-rendered indicator can create a technical problem, as illustrated in FIGS. 2A and 2B. The problem arises from the geometric ambiguity of determining the input’s depth when the input 104 is not intersecting any virtual objects. As shown, when the input traverses the void space between objects, the system has no direct data from a surface intersection to ground the input’s position. The lack of a definitive surface reference means the input’s depth is undefined, which can lead to discontinuous rendering artifacts as the input moves between objects at different depths.

[0030] FIG. 2A illustrates the 3D space of the XR environment 102 in an example where a user is controlling a position of an input, rendered as indicator X, using a gaze-based signal. In the illustrated example there are four virtual objects, labeled 1-4. FIG. 2A shows the virtual objects in a relative coordinate frame where the Z-axis is defined by the direction of the user’s input 104. The depth of each virtual object is its distance along the Z-axis with respect to the input's origin. In the illustrated example, the input is in a void space between virtual objects, and a corresponding indicator may not be visible to the user.

[0031] FIG. 2B illustrates another example of the 3D space of the XR environment 102 where a user is controlling a position of an input, rendered as indicator X, using a hand-based signal. In the illustrated example there are four virtual objects, labeled 1-4. FIG. 2B shows the virtual objects in a relative coordinate frame where the Z-axis is defined by the direction of the input 104'. The depth of each virtual object is its distance along the Z-axis with respect to the input's origin, which corresponds to the location of the user’s hand. In the illustrated example, the input is in a void space between virtual objects, and a corresponding indicator may not be visible to the user.

[0032] For both of the examples shown in FIGS. 2A and 2B, as a user moves the input between the virtual objects, the depth of the input may be determined by a combination of the depths of adjacent virtual objects. For example, in the position shown in FIGS. 2A and 2B, the input is located between virtual objects 2, 3, and 4. In some examples, a depth for the input may be determined according to the depths of the nearest virtual objects. A variety of combination techniques may be used, including calculating a weighted average.

[0033] FIG. 3 illustrates an aspect of the technical solution, which involves reducing the 3D spatial relationship problem to a 2D geometric analysis. A technical advantage of the reduction is that the reduction enables real-time computation with a reduced number of processor cycles on resource-constrained hardware. The system accomplishes the reduction by computationally constructing a 2D plane 300 that is oriented to be perpendicular to the input 104. The orientationAtty Docket No. 0120-1111WO1 provides a consistent and normalized frame of reference for analyzing the proximity of neighboring objects, regardless of the input’s absolute direction in the 3D environment.

[0034] The process of creating the 2D representation involves a projection of the relevant 3D world coordinates, specifically the point where the input pierces the plane (X) and the closest points on the surfaces of nearby virtual objects (1, 2, 3, 4, 5), onto the newly constructed 2D plane. To identify the closest point on each virtual object, the system performs an iterative geometric calculation to find the 3D coordinates on the object's surface that have a minimum Euclidean distance to the infinite line defined by the input. The resulting 2D representation serves as the input for a subsequent geometric analysis, such as the generation of a Voronoi diagram, which is used to identify the nearest neighbors for the depth combination calculation. The process provides one non-limiting example of how to determine a depth for the input shown in FIGS. 2A and 2B.

[0035] As illustrated in FIG. 3, an example tessellation of the 2D plane may be defined, also referred to as a Voronoi diagram, in which the 2D plane is divided into polygon-shaped regions or cells. The cells are polygonal regions that collectively tile the 2D plane without gaps or overlaps, with each virtual object and the input being associated with a corresponding cell. A cell is a data structure representing a spatial partition, and the size and contour of a given cell are determined as those portions of the 2D plane that are closer to a given point (the virtual object or input within the cell) than to any of the other points. The primary purpose of generating the cells is to identify topological relationships, such as adjacency, between the set of points in the plane.

[0036] In an example, the system generates a Voronoi diagram to identify the nearest neighbors of the input. To determine the input’s interpolated depth, the system then determines a weight for each neighbor to perform a weighted average calculation of the depths of the neighbors. For example, the system may calculate Laplace weights, where the weight for a neighbor is determined by the ratio of the length of a shared Voronoi cell edge to the distance between the input and that neighbor. The system then uses the calculated weights to perform a natural neighbor interpolation, in which the weighted average of the depths of the nearest virtual objects is determined. In the illustrated example of FIG. 3, VO2-VO4 are the nearest neighbors to the input's location and are used to determine an interpolated depth for the input, whereas VO1 is not used because the cell for VO1 does not share an edge with the cell for X.

[0037] In some examples, the weights for the virtual objects may be determined according to a ratio of an area of a given neighbor cell in a tessellation that does not include the input's location,Atty Docket No. 0120-1111WO1 to an area of the cell of the input in a tessellation that does include the input's location, for example, using Sibson weights. In an example, a weight may be determined for each nearest neighbor virtual object, where a higher weight indicates the virtual object has a greater influence on the determined depth of the input than other virtual objects with lower weights.

[0038] After determining the input's 3D position through interpolation, a separate but related technical challenge involves deciding when to render the corresponding visual indicator. FIG. 4 provides a detailed illustration of the technical implementation for the input visualization logic, focusing on the concept of a visibility area. The visibility area is not a rendered object but a programmatic, non-visible buffer zone or boundary 402 defined at a configurable threshold distance (t) from the perimeter of a virtual object (VO1). The threshold can be a predetermined value, for example, a fixed distance of 300 pixels in screen space, that allows a system to trigger the rendering of an indicator before the input makes direct contact with an object’s actual geometry. The proactive visualization provides a visual affordance to a user.

[0039] A technical aspect of the visualization logic lies in the specific plane used for the distance calculation. The system measures the distances (dl, d2) between the input’s position (XI, X2) and the closest point on the boundary of the visibility area within a 2D plane that is aligned with a user’s viewpoint, not the plane perpendicular to the input that is used for depth interpolation. A technical advantage of the technical choice to use the viewpoint-aligned plane is that the choice decouples the indicator’s visualization trigger from the 3D depth of the virtual object. By performing the check in the viewpoint plane, the system can have an indicator appear at a perceptually consistent distance from an object’s edge, regardless of whether that object is near or far away in the 3D scene, creating predictable user interface behavior.

[0040] FIG. 4 illustrates the principle with two example positions of the input, XI and X2, relative to virtual object VO1. The figure is illustrated from a user’s viewpoint and shows an example input position XI where a corresponding indicator is not rendered because the input is located in a void space outside the visibility area defined by the boundary 402. In contrast, at position X2, the input is within the threshold distance, t, and intersects the visibility area. In response to the intersection, a corresponding indicator is rendered so that a user can see the indicator’s location with respect to the virtual object, which facilitates more precise interaction with the object.Atty Docket No. 0120-1111WO1

[0041] When an input is near multiple virtual objects, the visualization logic includes first selecting a single virtual object from a group of nearby objects to serve as the primary reference for the proximity check. The selected virtual object is identified based on the weights calculated during the depth interpolation process described with reference to FIG. 3. For example, in an implementation using Laplace weights, the virtual object having the highest calculated weight may be selected. A technical aspect of the system is the repurposing of the weights for the visualization logic, which is a design choice that conserves computational resources, as the system avoids performing a separate analysis to identify the primary reference object.

[0042] A specific logic path is also implemented to handle active user interactions. A technical challenge arises if a user initiates an interaction, such as a drag operation, and then moves the input beyond the visibility threshold. Hiding the indicator in that context would break the interaction. The technical solution is for the system to determine when an interaction state is active, for example, by detecting a sustained selection input. In response to detecting an active interaction state, the system maintains rendering of the indicator by overriding the standard distance-based visualization logic. The indicator is then hidden only after the system detects the completion of the interaction, for example, by receiving a signal corresponding to a release of the selection.

[0043] FIG. 5 is a flowchart of an example process 500 for determining a depth of an input in a virtual environment, such as an AR, MR, XR, or VR environment. The process 500 may be performed using HMD 11 and XR application 120 shown in at least FIG. 1.

[0044] The process 500 begins at block 510, where the system determines a user’s current input. The inputs for the step are real-time sensor data from the HMD 11 or other input devices, such as an eye-tracking camera or an inertial measurement unit (IMU). The computational action involves processing the sensor data to generate an input vector, which is the primary output of the step. The vector is defined by two pieces of data: a 3D origin point (e.g., 106 or 106’) and a 3D direction, which together define the input for interaction within the virtual space.

[0045] At block 520, the system identifies all nearby virtual objects and calculates the closest point on the surface or perimeter of each object to the input identified in the previous step. The inputs are the input vector and the set of all interactable virtual objects currently in the scene. To ensure efficient operation in complex scenes, the system may first query a spatial partitioning data structure, such as a k-d tree or an octree, to rapidly identify a small subset of potentially relevant virtual objects near the input, avoiding a costly search through every object in the scene. For eachAtty Docket No. 0120-1111WO1 virtual object in the candidate set, which is typically represented as a polygonal mesh, the system then performs a geometric calculation to find the 3D coordinates of the point on its surface that is geometrically closest to the infinite line defined by the input. One method for the calculation involves iterating through each triangle of the object’s mesh. For each triangle, the system computes the closest point on the plane of that triangle to the input line. If the projected point lies inside the triangle’s boundaries, that point is a candidate. If not, the closest point will lie on one of the triangle’s three edge segments. The system compares the candidate points from all triangles to identify the one with the minimum Euclidean distance to the input line. The output is a set of 3D coordinate points, with each point corresponding to a specific nearby virtual object.

[0046] At block 530, the system projects the 3D spatial problem into a 2D representation that can be processed with fewer computational resources. The inputs for the step are the set of 3D points from block 520 and the input vector from block 510. The computational action is the construction of a new 2D coordinate system on a plane (e.g., plane 300) that is oriented to be perpendicular to the input’s direction. The system then projects the 3D coordinates of the nearby virtual object points and an estimated location of the input onto the plane. The output is a set of 2D coordinate points that represent the relative lateral positions of the objects from the perspective of the input.

[0047] The mathematical transformation is achieved by creating a new, temporary orthonormal basis. The basis is a set of three mutually perpendicular unit vectors (U, V, W), where the W vector is defined to be parallel to the input’s direction vector. The U and V vectors consequently define the 2D plane that is perpendicular to the input. To project a 3D point from the world coordinate system into the new plane, the system represents the point’s coordinates in the new (U, V, W) basis. The projection is finalized by discarding the W coordinate, which corresponds to the depth along the input’s direction, leaving only the U and V coordinates that define the point’s 2D position. The input’s location projected onto the plane is an estimate; in some implementations, the location is the final interpolated position calculated in the preceding frame, which serves as the input for the current frame’s geometric analysis.

[0048] At block 540, the system calculates the input’s new depth. The inputs for the step are the 2D coordinate points from block 530 and the original 3D depth values of the corresponding virtual objects. While the conceptual output is a Voronoi diagram, an implementation that can reduce computational load may leverage the geometric duality between Voronoi diagrams and DelaunayAtty Docket No. 0120-1111WO1 triangulations. The system may first compute the Delaunay triangulation of the 2D projected points, which connects the points into a mesh of triangles such that no point lies inside the circumcircle of any triangle. The Voronoi diagram can then be derived directly from the triangulation, as the vertices of the Voronoi cells correspond to the circumcenters of the Delaunay triangles. The two-step process can be faster than computing the Voronoi diagram directly.

[0049] Once the nearest neighbors are identified (as the objects whose Voronoi cells share an edge with the input’s cell), the system calculates a weight for each neighbor. In some implementations, the calculation is achieved using Laplace weights in a form of barycentric coordinate calculation where the weight for a given neighbor i is determined by the formula weight_i = (length_i / distance_i). The formula has specific geometric significance: the formula assigns greater influence not only to neighbors that are physically close (where distance_i is small) but also to those that topologically “wrap around” the input’s projected position more significantly (where the shared Voronoi edge length_i is large), indicating a stronger geometric relationship.

[0050] The use of Laplace weights contrasts with other interpolation schemes, such as inverse distance weighting (IDW). An IDW method would only consider proximity (distance_i) and would ignore the geometric arrangement and topological influence of the neighbors. By incorporating both distance and shared boundary length, the Laplace weighting method can produce a stable interpolation result.

[0051] After calculating the weights for each nearest neighbor, the system performs the final step of the depth determination. The final step involves calculating a weighted average of the neighbors’ original 3D depths using their respective calculated weights. The use of a weighted average is one example of a mathematical combination that can be used to determine the input's depth; other combination functions could be used in other implementations. The final output is a single scalar value representing the newly interpolated depth of the input, which is then used to define the 3D position of the input for the current frame.

[0052] To provide a concrete example of the calculation in block 540, consider a scenario where the system identifies three nearest neighbors: VO2 at a depth of 1.0m, VO3 at a depth of 1.2m, and VO4 at a depth of 1. Im. After generating the Voronoi diagram, the system calculates a Laplace weight for each neighbor based on its geometric relationship to the input. In the hypothetical case, VO2 is closest and has the longest shared edge, resulting in a weight of 0.5; VO4 is further away, with a weight of 0.3; and VO3 is the furthest of the three, with a weight of 0.2. The system thenAtty Docket No. 0120-1111WO1 calculates the input’s new depth as a weighted average: (1 ,0m 0.5) + (1 ,2m 0.2) + (1 ,1m * 0.3) = 0.5m + 0.24m + 0.33m = 1.07m. The result is a smooth, deterministic depth that is logically situated between the surrounding objects.

[0053] The methods for determining input depth disclosed herein provide several technical aspects that affect the functioning of the XR system. For example, using a weighted average of nearest neighbors can result in a visual indicator corresponding to the input moving along a smooth trajectory towards a target. The methods can produce smooth transitions when the input moves between non-overlapping virtual objects, even when the objects are at substantially different depths. A further technical aspect is realized when using a hand-based input, as grounding the depth measurement to the input’s origin rather than the user’s head maintains a consistent geometric reference frame between the origin of the input and the calculated depth.

[0054] In some implementations, the technical solution provides a further technical aspect by modifying the data provided to the rendering pipeline during input transitions. In addition to interpolating the input’s depth, the system may also rotate an orientation of a corresponding indicator. The computational process for the feature involves, first, retrieving orientation data for both a starting virtual object and a target virtual object (e.g., a selected virtual object having a highest weight). The orientation data is typically stored as a normal vector or a quaternion representing the object’s surface orientation at the closest point. As the system iteratively calculates the input’s interpolated depth for each frame of the transition, the system simultaneously interpolates the indicator’s orientation between the starting orientation and the target orientation. In a non-limiting example, the smooth rotation can be achieved using a Spherical Linear Interpolation (SLERP) function on the quaternions, which calculates the shortest path on the surface of a four-dimensional sphere and ensures a constant angular velocity for the rotation. Upon completion of the transition, the indicator’s orientation is aligned with the orientation of the second virtual object the indicator is approaching. The rotational alignment provides a continuous orientation vector to the rendering system, resulting in a visual output that reflects the surface normal of the impending target.

[0055] FIG. 6 is a flowchart of an example process 600 for controlling a visualization of an input in a virtual environment, such as an AR, MR, XR, or VR environment. The process 600 may be performed using HMD 11 and XR application 120 shown in at least FIG. 1.Atty Docket No. 0120-1111WO1

[0056] The process 600 begins at block 610, where the system selects a virtual object from the set of nearest neighbors identified in process 500. The inputs for the step are the list of nearest neighbor objects and their corresponding calculated weights. The computational action is a comparison to find the object that will serve as the primary reference for the visualization logic. In a non -limiting example, the system may select the virtual object having the highest calculated weight. The output is the identifier of the single selected virtual object.

[0057] At block 620, the system determines a distance between the input and the selected virtual object. The calculation is performed in a second plane, which is a 2D plane that corresponds to the user’s viewpoint, not the plane perpendicular to the input that was used for the depth interpolation in process 500. The inputs are the 2D screen-space coordinates of the selected virtual object and the 2D screen-space coordinates of the input. The computational action is to calculate the Euclidean distance between the two points. The output is a single scalar value, d, representing the perceived visual distance between the input and the object.

[0058] At block 630, the system makes a determination of whether to render the input as an indicator. The inputs are the distance d from block 620 and a predefined visibility threshold t. The computational action is a conditional check to determine if the distance is meeting the threshold (e.g., if d is less than t). The output is a binary decision that is passed to a rendering engine to either render the indicator at the input's calculated 3D position or keep the indicator hidden.

[0059] As previously introduced, the system is further configured with specific logic to handle active user interactions. To prevent a disorienting experience where an indicator disappears during a drag operation, the system first determines if an interaction is active. In response to determining an active interaction, the system maintains rendering of the indicator, overriding the standard distance-based visualization logic and keeping the indicator visible even when the distance exceeds the threshold. The indicator is then hidden only after the user completes the interaction.

[0060] To provide a concrete example for the decision at block 630, the visibility threshold t may be configured to a fixed screen-space distance, such as 300 pixels. If the system, at block 620, calculates the distance d between the input and the selected virtual object to be 250 pixels, the comparison 250 < 300 evaluates to true. Consequently, the system renders the indicator. In a subsequent frame, if the user moves the input further away and the calculated distance d becomes 350 pixels, the comparison 350 < 300 evaluates to false, and the indicator remains hidden.Atty Docket No. 0120-1111WO1

[0061] The system also includes logic to handle a specific technical challenge that arises when a user moves an input between two virtual objects that are very close to each other. In such a scenario, the input can rapidly exit the visibility area of the first object and enter the visibility area of the second object, which can cause the rendering pipeline to rapidly alternate the indicator's visibility state, producing a flickering artifact. A technical solution to prevent the rendering artifact is for the system to determine if the visibility area of the selected virtual object overlaps the visibility area of a second, adjacent virtual object. In response to determining an overlap, the system maintains a continuous rendering state for the indicator as the user transitions the input between the two objects.

[0062] To cohesively manage the different operational contexts described — such as direct intersection, interpolation in a void, and inactivity — the input positioning system can be implemented as a state machine. The architectural approach allows the system to efficiently manage computational resources and provide a responsive user experience by transitioning between three distinct operational states: an intersecting state, an interpolating state, and an inactive state.

[0063] The system is in the intersecting state when the user’s input is directly intersecting a collider of a virtual object. In the intersecting state, the complex interpolation logic of process 500 is bypassed entirely. The input’s depth is determined directly by the 3D coordinates of the intersection point on the object’s surface, and a corresponding indicator is rendered as visible. The state provides a high degree of precision and responsiveness when the user is actively directing the input toward an object.

[0064] The interpolating state is activated when the input is traversing a void but is within a predefined proximity to one or more virtual objects. In the interpolating state, the system executes the depth interpolation (process 500) and visualization logic (process 600) on each frame to calculate the input’s position and determine the visibility of a corresponding indicator. The state is responsible for creating the smooth, predictable trajectory of the input through empty space.

[0065] The inactive state is a power-saving and performance-optimizing state that occurs when the input is directed into a large void, far from any virtual objects. Specifically, the inactive state is entered when an initial query for nearby objects (block 520) returns an empty set. A technical aspect of the inactive state is that both the interpolation and visualization logic can be suspended,Atty Docket No. 0120-1111WO1 and a corresponding indicator remains hidden. The state prevents the system from performing unnecessary calculations when the user is not pointing near any interactable content.

[0066] The transitions between the operational states are governed by specific triggers. The system transitions from the intersecting state to the interpolating state at the moment an input intersection check changes from TRUE to FALSE, for instance, when the user moves the input off an edge of a virtual object’s collider. Conversely, a transition from the interpolating state back to the intersecting state occurs when the input subsequently intersects a physical collider. A transition from the interpolating state to the inactive state is triggered when the set of nearby neighbor objects becomes empty. Finally, the system transitions from the inactive state back to the interpolating state as soon as at least one nearby object is detected by the initial proximity check, at which point the interpolation calculations resume.

[0067] The system is also configured to handle specific edge cases by bypassing the interpolation logic to ensure predictable behavior. If the input is directed into a large void where no virtual objects are identified as being nearby (i.e., the set of neighbors from block 520 is empty), the system enters the inactive state, the interpolation process 500 does not run, and a corresponding indicator remains hidden. Conversely, if a user directs the input toward a dense cluster of virtual objects that are overlapping from the user’s perspective, the interpolation logic is also bypassed. In that scenario, the system reverts to a direct intersection model, and the input’s depth will jump immediately to the surface of the object being hit, with a corresponding indicator remaining visible as the input traverses between the overlapping object colliders.

[0068] The processes 500 and 600 are not performed a single time but can be executed iteratively in a continuous loop, for example, at a high frequency such as 60Hz. The iterative execution allows the input’s position and the visibility of a corresponding indicator to be updated in real-time in response to user signals.

[0069] To implement the state-driven logic described above, an overall system architecture may be structured into several distinct logical components or software modules. In some implementations, the XR application 120 includes an input manager, a scene graph manager, an interpolation engine, and a rendering engine that work in concert to perform the functions disclosed herein.

[0070] The input manager is responsible for interfacing with the input / output (I / O) devices 740 to receive and process raw sensor data, such as orientation data from an IMU or image data fromAtty Docket No. 0120-1111WO1 eye-tracking cameras. The input manager processes the raw sensor data through one or more digital fdters, such as a Kalman fdter or an exponential moving average, to reduce noise and produce a stable signal. The input manager then transforms the fdtered data into a canonical computational input. The standardized input, which is defined by a 3D origin and direction, is the primary output of the input manager and serves as a data input for the other modules.

[0071] The scene graph manager maintains the hierarchical data structure, such as a k-d tree or octree, that contains the spatial data for all virtual objects in the environment. The function of the scene graph manager is to respond to spatial queries from the interpolation engine. To process a query, the scene graph manager traverses its tree data structure, performing intersection tests between the input line and the bounding volumes of the tree’s nodes. By recursively checking only the nodes that the input intersects, the scene graph manager efficiently culls the scene to return a list of candidate virtual objects that are in the general proximity of the input, which avoids a costly linear search of every object in the scene.

[0072] The interpolation engine is the core logical module that implements the functionality of process 500 and process 600. The interpolation engine takes the input from the input manager and the list of candidate objects from the scene graph manager as data inputs. The interpolation engine then performs the geometric projection, nearest neighbor identification, and weighted average calculations to determine the input’s final 3D position and a corresponding visibility status. The output of the interpolation engine is a 3D coordinate and a binary visibility flag.

[0073] The rendering engine receives the final 3D position and the visibility flag from the interpolation engine. The rendering engine, which may run on a graphics processing unit (GPU), is responsible for the final visual output. The rendering engine takes the calculated position and draws the geometry for a corresponding indicator at that location in the 3D scene. If the visibility flag is false, the rendering engine does not draw the indicator. Finally, the rendering engine composites the indicator into the final frame that is sent to the display.

[0074] During a single, real-time update cycle, for example, one frame in a 60Hz system, the software modules interact in a specific sequence. The cycle begins with the input manager processing the latest sensor data to generate an updated input. The generated input is passed to the interpolation engine. The interpolation engine immediately sends a spatial query, containing the input data, to the scene graph manager, which returns a list of nearby virtual objects. Using the returned list, the interpolation engine executes the core interpolation and visualization logic,Atty Docket No. 0120-1111WO1 calculating the input’s new 3D position and a ‘TRUE’ visibility flag. The resulting 3D position and visibility flag are then sent to the rendering engine. The rendering engine uses the data to draw a visible indicator into the scene for the frame that is about to be displayed to the user.

[0075] FIG. 7 is a block diagram illustrating an example computing system 700 that can be used to implement the operational architectures, processes, and scenarios disclosed herein. In some examples, the computing system 700 represents the HMD 11 described in FIG. 1. The computing system 700 is configured to perform the input position and visualization tasks described in the preceding figures.

[0076] As depicted in FIG. 7, the computing system 700 includes a processing system 710, a storage system 720, a communication interface 730, and one or more VO devices 740. The processing system 710 is operatively linked to the communication interface 730, the one or more VO devices 740, and the storage system 720. The computing system 700 can further include other components, such as a power source and an enclosure, that are not shown for clarity. For mobile implementations such as the HMD 11, the power source is typically a battery, making the device a battery-powered device.

[0077] The processing system 710 includes circuitry that retrieves and executes operating software from the storage system 720. In resource-constrained devices like the HMD 11, the processing system 710 may be a System-on-Chip (SoC). An SoC integrates multiple processing components onto a single die to manage power consumption. Such components can include one or more central processing units (CPUs) that function as general -purpose processor cores for the operating system and application logic, and a GPU for rendering the XR environment 102 and performing parallelized computations. The processing system 710 can also include specialized processors, such as an image signal processor (ISP) or a digital signal processor (DSP), which can be designed to handle real-time sensor data from VO devices 740 with low power consumption.

[0078] The one or more VO devices 740 can include peripherals that facilitate interaction between a user and the computing system 700 or that allow the system to receive input from the environment. In implementations where the computing system 700 represents the HMD 11, the VO devices 740 include the display 10 for presenting the XR environment 102, and sensors for detecting user actions. The sensors may include an IMU for determining head motion and orientation, eye-tracking cameras for determining gaze direction to generate a gaze-based input,Atty Docket No. 0120-1111WO1 and world-facing or depth-sensing cameras for determining hand position and orientation to generate a hand-based input.

[0079] The storage system 720 can include volatile and nonvolatile, removable, and nonremovable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. The storage system 720 can be implemented as a single storage device or across multiple storage devices or sub-systems. Examples of storage media include random access memory (RAM), read-only memory (ROM), magnetic disks, optical disks, and flash memory. In some implementations, the storage media can be non -transitory.

[0080] The processing system 710, which includes at least one electronic processor, is typically mounted on a circuit board that also holds the storage system 720, which functions as a memory and a non-transitory computer-readable medium. The memory of the storage system 720 is communicably coupled to the electronic processor of the processing system 710. The memory stores executable instructions in the form of operating software, which includes computer programs, firmware, or some other form of machine-readable program instructions. As described herein, when the executable instructions are loaded from the memory into the electronic processor and executed, the instructions cause the system to perform the specific, ordered set of operations disclosed throughout the specification. The execution of instructions physically configures the general-purpose circuitry of the electronic processor, transforming the processor into a special - purpose machine specifically designed to solve the identified technical problem of determining an input’s depth and the visibility of a corresponding indicator when traversing void spaces in a 3D environment.

[0081] The execution of the instructions is a concrete, physical process that transforms the computing system 700. For example, to perform the disclosed method, the processing system 710 fetches a sequence of instructions from the storage system 720. A control unit within the processor decodes the instructions, which in turn physically configure and direct an arithmetic logic unit (ALU) or the GPU to perform specific mathematical operations on numerical values representing spatial coordinates. Specifically, one set of instructions directs the processor to fetch numerical values representing the 3D coordinates of nearby virtual objects and the current input from specific memory addresses. The processor then projects the 3D coordinates into a 2D plane and performs a series of iterative geometric calculations to generate a Voronoi diagram. Another set ofAtty Docket No. 0120-1111WO1 instructions directs the processor to calculate Laplace weights by performing a division operation on the length of shared Voronoi edges and the distances to each neighboring object. The sequence of physical hardware operations, dictated by the stored software, transforms the general -purpose processing system 710 into a special-purpose machine that generates a technically altered input position, resulting in a smooth and predictable trajectory for rendering on the display 10.

[0082] In some implementations, the specific sequence of operations described constitutes an inventive concept that provides a tangible, technical result and amounts to more than the abstract idea of interpolation itself. The output of the process is a specific, calculated 3D coordinate for the input’s position in void space. The coordinate is a technically altered data structure that enables the rendering engine to overcome the specific technical problems of rendering failures and unpredictable output that are inherent in conventional XR systems. By generating the specific, technically altered data, the system provides a specific improvement to the functioning of the computer itself, transforming the human-computer interface from an unstable and imprecise tool into a stable and predictable navigation system. The recited improvement is not a generic application of a mathematical formula but a specific technological solution to a problem that arises directly from the nature of 3D computer-generated environments.

[0083] The disclosed improvements are inextricably tied to the specific technological environment of XR and are not a disembodied mathematical concept. The problem being solved, a data deficit that causes system instability when an input traverses a void, is a problem that exists only within 3D computer-generated environments. The disclosed solution is a specific, practical application that leverages a computer’s ability to perform high-speed geometric calculations on data representing a virtual space. The solution improves the functioning of the computer itself by transforming the human-computer interface from a system that fails under common operating conditions into one that provides a stable, predictable, and efficient means of navigation. By proactively generating depth data where none existed, the system is fundamentally altered from a reactive device that depends on direct intersection into a predictive one that can maintain a stable operational state in a void, an improvement unique to the technological context.

[0084] The operating software includes instructions that direct the processing system 710 to perform the steps of the disclosed methods, orchestrating a specific data flow and division of labor between the CPU and GPU. The process begins when the CPU handles the serial task of integrating sensor data from the I / O devices 740 to determine the initial input vector. The CPU then retrievesAtty Docket No. 0120-1111WO1 the 3D coordinates of nearby virtual objects from the storage system 720. For the depth interpolation (process 500), the computationally intensive and highly parallelizable task of generating the Voronoi diagram from the 2D projected points is offloaded to the GPU, which is architecturally optimized for such rapid geometric calculations. Once the GPU completes the interpolation and visibility calculations, the final 3D position of the input is passed to the GPU’s rendering pipeline to generate the next frame for the display. The operating software can include instructions that, when executed, cause the computing system 700 to perform the steps of the process 500 described with reference to FIG. 5 for determining input depth, and the process 600 described with reference to FIG. 6 for determining indicator visualization. In implementations utilizing a GPU, the computationally intensive tasks of generating the Voronoi diagram and calculating weights may be offloaded to the GPU for parallel processing to manage performance and power draw, while the CPU manages the application logic and sensor input.

[0085] The disclosed methods can be applied to a variety of input modalities beyond gaze and hand signals. For example, an input can be defined by the physical position and orientation of a six-degrees-of-freedom (6-DOF) controller. In another implementation, a 2D input device such as a trackpad or mouse can be used for indirect control, where the 2D input coordinates are mapped directly onto the 2D plane used for the interpolation calculations to determine the 3D position of a corresponding indicator. Furthermore, the rendering of an indicator can be more nuanced than a binary on / off state. In some implementations, a system can modify an opacity of an indicator to create a gradient based on the indicator's distance to a selected virtual object, causing the indicator to fade in smoothly by increasing opacity as the indicator approaches the visibility threshold. The shape of the visibility area itself can also be non-uniform, dynamically adapting to the specific geometry of the virtual object to provide a more context-aware interaction boundary.

[0086] The disclosed system provides specific technical benefits in various applications. In professional ‘Computer-Aided Design (CAD)’ applications, the system enables users to precisely select and manipulate sparsely located control points or vertices on a complex 3D model without a corresponding indicator getting lost in void space. For ‘Commercial / Retail’ applications, a user navigating a virtual showroom can smoothly target products on shelves at different depths, with the system providing a clear path that enhances spatial understanding. In ‘System-Level User Interfaces’, the disclosed methods provide utility for reliably interacting with system notifications, menus, and application windows that a user can place anywhere in a 3D personal computing space.Atty Docket No. 0120-1111WO1

[0087] Furthermore, a system’s operational frequency can be synchronized with the sampling rate of the input sensors that generate the user signal, such as eye-tracker cameras or an IMU. The iterative nature of the disclosed methods, for example, running at 60Hz or 90Hz to match a display’s refresh rate, can use a steady stream of high-fidelity input data. A technical challenge can arise from potential mismatches in the frequencies; for instance, sensor data may arrive less frequently than frames are rendered. To handle such cases and ensure a smooth visual update rate, a system may employ predictive filtering or motion extrapolation algorithms. The algorithms estimate an input’s likely position and orientation between actual sensor readings, allowing the interpolation and rendering pipeline to continue operating at the target frame rate without perceptible stutter or lag.

[0088] Building on the need for real-time operation, one optimization involves the geometric data used for proximity calculations. For complex scenes, a rendering mesh of a virtual object may have a very high polygon count to achieve a detailed visual appearance. Performing the ‘closest point’ calculation (block 520) against the high-polygon-count mesh on every frame can be computationally intensive. To reduce the computational load, a system may instead perform the geometric queries against a simplified collision proxy that has a lower polygon count than the rendering mesh. A technical aspect of the optimization is the reduction in polygon count, which directly reduces the number of calculations required for the geometric tests and enables the interpolation logic to run consistently at a high frame rate on resource-constrained devices.

[0089] In addition to optimizing the geometric data, the core interpolation algorithm itself can be varied. While the Voronoi-based natural neighbor interpolation described provides a smooth trajectory for a corresponding indicator, other interpolation algorithms may be used. In some implementations, an inverse distance weighting (IDW) method can be employed. In an IDW implementation, the weight for each neighboring virtual object is calculated based solely on the inverse of the object's distance to the input in the 2D plane. The IDW approach presents a technical trade-off IDW is computationally less expensive as the approach does not require the generation of a Voronoi diagram, and the approach produces a CO continuous indicator trajectory. In other implementations, Sibson interpolation may be used. Sibson interpolation calculates weights based on the ratio of overlapping areas of Voronoi cells and can produce a Cl continuous path at a potentially higher computational cost.Atty Docket No. 0120-1111WO1

[0090] The real-time operation of any of the disclosed methods can be facilitated by the coordinated performance of the hardware components within the computing system 700, which in some cases is managed by a real-time operating system (RTOS). An RTOS is designed to process data as the data comes in, without buffering delays, a factor in XR interactions where a missed processing deadline can result in a rendered indicator that lags or appears jittery.

[0091] It is also to be understood that although certain drawings illustrate hardware and software located within particular devices, the presented depictions are for illustrative purposes only. In some implementations, the illustrated components may be combined or divided into separate software, firmware, or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how the components are combined or divided, hardware and software components may be located on the same computing device or may be distributed among different computing devices connected by one or more networks or other suitable communication links.

[0092] Moreover, various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed hardware such as application-specific integrated circuits, computer hardware, firmware, software, or combinations thereof. The various implementations can include implementation in one or more computer programs that are executable or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0093] Computer programs (also known as programs, software, software applications, or code) include computer readable or machine instructions for a programmable electronic processor and can be implemented in a high-level procedural or object-oriented programming language, or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions or data to a programmable processor.Atty Docket No. 0120-1111WO1

[0094] The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some implementations, a computer program includes one sequence of instructions. In some implementations, a computer program includes a plurality of sequences of instructions. In some implementations, a computer program is provided from one location. In other implementations, a computer program is provided from a plurality of locations. In various implementations, a computer program includes one or more software modules. In various implementations, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.

[0095] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosed subject matter belongs. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0096] To support user control over data, implementations may provide users with notice regarding data use and tools to manage user information. For example, where a user provides consent for the analysis of information related to system usage or for the collection of information about preferences or activities, the user may be provided with one or more interfaces or tools. The provided tools can allow the user to manage settings and make elections regarding when certain information is collected. In some implementations, these tools may provide options to review, modify, or delete information associated with the user's usage of the system. In addition, in some cases, certain data may be processed in one or more ways before the data is stored or used. For instance, data may be aggregated with data from other users, or location information may be generalized so that a specific location of a user cannot be determined. The data processing allows the user to maintain control over what information is collected and how that information is used.

[0097] A number of implementations have been described. It will be understood that various modifications, including the separation or integration of system components, may be made without departing from the spirit and scope of the disclosure. It should be understood that various alternatives to the implementations described may be employed, and the appended claims are intended to cover such variations and changes as fall within the spirit of the described subject matter.Atty Docket No. 0120-1111WO1

[0098] Clause 1 . A computer-implemented method, comprising: identifying an input in a virtual space (e.g., XR environment), the input defined by an origin and a direction; determining a location of the input in a plane (e.g., virtual and / or invisible plane, e.g., in the virtual space); identifying at least one virtual object (e.g., in the virtual space) based on at least one criterion associated with the location of the input in the plane; and determining a depth (e.g., from a user, from eyes of the user and / or from the origin of the input) of the input based on a depth (e.g., from the same reference as for the input) of the at least one virtual object. Then, an indicator may be rendered at a 3D position of the input at the depth of the input.

[0099] Clause 2. The method of clause 1, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects in the virtual space, and determining the depth of the input by calculating a combination of depths of a subset of the plurality of virtual objects.

[0100] Clause 3. The method of clause 2, wherein calculating the combination of depths includes: partitioning the plane into a plurality of cells, the plurality of cells including an input cell associated with the input and a plurality of object cells, the plurality of object cells respectively associated with virtual objects of the subset; and determining weights for the virtual objects of the subset, wherein a weight for a virtual object of the subset is determined according to a length of a shared boundary between the input cell and an object cell of the plurality of object cells associated with the virtual object..

[0101] Clause 4. The method of clause 3, wherein determining the weight for the virtual object of the subset further includes determining a ratio of the length of the shared boundary to a distance between the input and the virtual object in the plane.

[0102] Clause 5. The method of clause 3 or 4, further comprising: selecting a virtual object from the subset based on the determined weight; determining a distance in a second plane between the selected virtual object and the input, the second plane optionally corresponding to a viewpoint; and rendering the input as an indicator in response to the determined distance meeting (e.g., being smaller than or being equal to) a threshold.

[0103] Clause 6. The method of clause 5, further comprising: determining an interaction with the selected virtual object; and in response to determining the interaction, maintaining rendering of the indicator when the determined distance exceeds the threshold.

[0104] Clause 7. The method of any one of clauses 2 to 6, wherein the subset is identified based on a proximity of the virtual objects in the subset to the location of the input in the plane.Atty Docket No. 0120-1111WO1

[0105] Clause 8. The method of any one of clauses 2 to 7, further comprising rotating an orientation of an indicator representing the input during a transition between a first virtual object and a second virtual object of the subset, wherein the orientation of the indicator is aligned with an orientation of the second virtual object upon completion of the transition.

[0106] Clause 9. The method of any one of clauses 1 to 8, wherein the at least one criterion includes a proximity of the at least one virtual object to the location of the input in the plane.

[0107] Clause 10. The method of any one of clauses 1 to 9, wherein the at least one criterion is an intersection between the input and the at least one virtual object, and determining the depth of the input includes setting the depth to a point of the intersection on the at least one virtual object.

[0108] Clause 11. The method of any one of clauses 1 to 10, wherein the input is identified based on a user signal generated by one or more sensors.

[0109] Clause 12. The method of clause 11, wherein the user signal corresponds to one of a detected eye gaze, a detected head gaze, or a detected hand position.

[0110] Clause 13. The method of any one of clauses 1 to 12, wherein determining the location of the input in the plane includes using a previously determined depth of the input to calculate the location.

[0111] Clause 14. The method of any one of clauses 1 to 13, wherein the plane is oriented to be perpendicular to the direction of the input.

[0112] Clause 15. A non-transitory computer-readable medium storing executable instructions that when executed by an electronic processor, cause the electronic processor to: identify an input in a virtual space, the input defined by an origin and a direction; determine a location of the input in a plane; identify at least one virtual object based on at least one criterion associated with the location of the input; and determine a depth of the input based on a depth of the at least one virtual object.

[0113] Clause 16. The non-transitory computer-readable medium of clause 15, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects and determining weights for a subset of the plurality of virtual objects, and the instructions further cause the electronic processor to: select a virtual object from the subset based on the determined weights; determine a distance in a second plane between the selected virtual object and the input, the second plane corresponding to a viewpoint; and render the input as an indicator in response to the determined distance meeting a threshold.Atty Docket No. 0120-1111WO1

[0114] Clause 17. The non-transitory computer-readable medium of clause 16, wherein the plurality of virtual objects are identified based on a proximity to the location of the input.

[0115] Clause 18. The non-transitory computer-readable medium of clause 16, wherein the instructions further cause the electronic processor to: determine an interaction with the selected virtual object; and in response to determining the interaction, maintain rendering of the indicator when the determined distance exceeds the threshold.

[0116] Clause 19. The non-transitory computer-readable medium of any one of clauses 16 to18, wherein the instructions further cause the electronic processor to maintain rendering of the indicator during a transition of the input between the selected virtual object and a second virtual object having an overlapping visibility area defined by the threshold.

[0117] Clause 20. The non-transitory computer-readable medium of any one of clauses 15 to19, wherein the at least one criterion is an intersection between the input and the at least one virtual object, and wherein the instructions that cause the electronic processor to determine the depth of the input further cause the electronic processor to set the depth to a point of the intersection on the at least one virtual object.

[0118] Clause 21. A system comprising: an electronic processor; and a memory communicably coupled to the electronic processor, the memory storing instructions that, when executed by the electronic processor, cause the system to: identify an input in a virtual space, the input defined by an origin and a direction; determine a location of the input in a plane; identify at least one virtual object based on at least one criterion associated with the location of the input; and determine a depth of the input based on a depth of the at least one virtual object.

[0119] Clause 22. The system of clause 21, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects and determining weights for a subset of the plurality of virtual objects, and the instructions further cause the system to: select a virtual object from the subset based on the determined weights; determine a distance in a second plane between the selected virtual object and the input, the second plane corresponding to a viewpoint; and render the input as an indicator in response to the determined distance meeting a threshold.

[0120] Clause 23. The system of clause 22, wherein the instructions further cause the system to: determine an interaction with the selected virtual object; and in response to determining the interaction, maintain rendering of the indicator when the determined distance exceeds the threshold.Atty Docket No. 0120-1111WO1

[0121] Clause 24. The system of clause 22 or 23, wherein the instructions further cause the system to maintain rendering of the indicator during a transition of the input between the selected virtual object and a second virtual object having an overlapping visibility area defined by the threshold.

[0122] Clause 25. The system of any one of clauses 21 to 24, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects in the virtual space, and the instructions that cause the system to determine the depth of the input further cause the system to calculate a combination of depths of a subset of the plurality of virtual objects, the calculation of the combination of depths including: partitioning the plane into a plurality of cells, the plurality of cells including an input cell associated with the input and a plurality of object cells, the plurality of object cells respectively associated with virtual objects of the subset; and determining a weight for a virtual object of the subset according to a length of a shared boundary between the input cell and an object cell of the plurality of object cells associated with that virtual object.

Claims

Atty Docket No. 0120-1111WO1WHAT TS CLAIMED TS;1. A method, comprising: identifying an input in a virtual space, the input defined by an origin and a direction; determining a location of the input in a plane; identifying at least one virtual object based on at least one criterion associated with the location of the input; and determining a depth of the input based on a depth of the at least one virtual object.

2. The method of claim 1, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects in the virtual space, and determining the depth of the input by calculating a combination of depths of a subset of the plurality of virtual objects.

3. The method of claim 2, wherein calculating the combination of depths includes: partitioning the plane into a plurality of cells, the plurality of cells including an input cell associated with the input and a plurality of object cells, the plurality of object cells respectively associated with virtual objects of the subset; and determining weights for the virtual objects of the subset, wherein a weight for a virtual object of the subset is determined according to a length of a shared boundary between the input cell and an object cell of the plurality of object cells associated with the virtual object.

4. The method of claim 3, wherein determining the weight for the virtual object of the subset further includes determining a ratio of the length of the shared boundary to a distance between the input and the virtual object in the plane.

5. The method of claim 3, further comprising: selecting a virtual object from the subset based on the determined weights; determining a distance in a second plane between the selected virtual object and the input, the second plane corresponding to a viewpoint; and rendering the input as an indicator in response to the determined distance meeting a threshold.Atty Docket No. 0120-1111WO16. The method of claim 5, further comprising: determining an interaction with the selected virtual object; and in response to determining the interaction, maintaining rendering of the indicator when the determined distance exceeds the threshold.

7. The method of claim 2, wherein the subset is identified based on a proximity of the virtual objects in the subset to the location of the input in the plane.

8. The method of claim 2, further comprising rotating an orientation of an indicator representing the input during a transition between a first virtual object and a second virtual object of the subset, wherein the orientation of the indicator is aligned with an orientation of the second virtual object upon completion of the transition.

9. The method of claim 1, wherein the at least one criterion includes a proximity of the at least one virtual object to the location of the input in the plane.

10. The method of claim 1, wherein the at least one criterion is an intersection between the input and the at least one virtual object, and determining the depth of the input includes setting the depth to a point of the intersection on the at least one virtual object.

11. The method of claim 1, wherein the input is identified based on a user signal generated by one or more sensors.

12. The method of claim 11, wherein the user signal corresponds to one of a detected eye gaze, a detected head gaze, or a detected hand position.

13. The method of claim 1, wherein determining the location of the input in the plane includes using a previously determined depth of the input to calculate the location.Atty Docket No. 0120-1111WO114. The method of claim 1, wherein the plane is oriented to be perpendicular to the direction of the input.

15. A non-transitory computer-readable medium storing executable instructions that when executed by an electronic processor, cause the electronic processor to: identify an input in a virtual space, the input defined by an origin and a direction; determine a location of the input in a plane; identify at least one virtual object based on at least one criterion associated with the location of the input; and determine a depth of the input based on a depth of the at least one virtual object.

16. The non-transitory computer-readable medium of claim 15, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects and determining weights for a subset of the plurality of virtual objects, and the instructions further cause the electronic processor to: select a virtual object from the subset based on the determined weights; determine a distance in a second plane between the selected virtual object and the input, the second plane corresponding to a viewpoint; and render the input as an indicator in response to the determined distance meeting a threshold.

17. The non-transitory computer-readable medium of claim 16, wherein the plurality of virtual objects are identified based on a proximity to the location of the input.

18. The non-transitory computer-readable medium of claim 16, wherein the instructions further cause the electronic processor to: determine an interaction with the selected virtual object; and in response to determining the interaction, maintain rendering of the indicator when the determined distance exceeds the threshold.Atty Docket No. 0120-1111WO119. The non-transitory computer-readable medium of claim 16, wherein the instructions further cause the electronic processor to maintain rendering of the indicator during a transition of the input between the selected virtual object and a second virtual object having an overlapping visibility area defined by the threshold.

20. The non-transitory computer-readable medium of claim 15, wherein the at least one criterion is an intersection between the input and the at least one virtual object, and wherein the instructions that cause the electronic processor to determine the depth of the input further cause the electronic processor to set the depth to a point of the intersection on the at least one virtual object.

21. A system comprising: an electronic processor; and a memory communicably coupled to the electronic processor, the memory storing instructions that, when executed by the electronic processor, cause the system to: identify an input in a virtual space, the input defined by an origin and a direction; determine a location of the input in a plane; identify at least one virtual object based on at least one criterion associated with the location of the input; and determine a depth of the input based on a depth of the at least one virtual object.

22. The system of claim 21, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects and determining weights for a subset of the plurality of virtual objects, and the instructions further cause the system to: select a virtual object from the subset based on the determined weights; determine a distance in a second plane between the selected virtual object and the input, the second plane corresponding to a viewpoint; and render the input as an indicator in response to the determined distance meeting a threshold.Atty Docket No. 0120-1111WO123. The system of claim 22, wherein the instructions further cause the system to: determine an interaction with the selected virtual object; and in response to determining the interaction, maintain rendering of the indicator when the determined distance exceeds the threshold.

24. The system of claim 22, wherein the instructions further cause the system to maintain rendering of the indicator during a transition of the input between the selected virtual object and a second virtual object having an overlapping visibility area defined by the threshold.

25. The system of claim 21, wherein identifying the at least one virtual object includes identifying a plurality of virtual objects in the virtual space, and the instructions that cause the system to determine the depth of the input further cause the system to calculate a combination of depths of a subset of the plurality of virtual objects, the calculation of the combination of depths including: partitioning the plane into a plurality of cells, the plurality of cells including an input cell associated with the input and a plurality of object cells, the plurality of object cells respectively associated with virtual objects of the subset; and determining weights for the virtual objects of the subset, wherein a weight for a virtual object of the subset is determined according to a length of a shared boundary between the input cell and an object cell of the plurality of object cells associated with the virtual object.

Images