Raycasting related to a hand in extended reality environments

Abstract

According to at least one implementation, a method includes defining a ray aligned along at least a portion of a hand of a user. The method further includes determining a modification to the ray based on a relationship between the ray and a portion of the user and displaying an indicator on a display of a wearable device based on the modification.

Description

Atty Docket No. 0120-1137WO1RAYCASTING RELATED TO A HAND IN EXTENDED REALITY ENVIRONMENTSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 735,678, filed December 18, 2024, the disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] An extended reality (XR) device includes a range of technologies that blend the physical and virtual worlds to create immersive experiences for a user. Such technologies can include virtual reality (VR) devices, which immerse users in a fully computer-generated environment; augmented reality (AR) devices, which overlay digital information onto the real world; and mixed reality (MR) devices, which merge real and virtual elements in an interactive way. XR devices are used in a variety of applications, including gaming, education, training, remote collaboration, and on-the-go computing.

[0003] To interact with content within an XR environment, a user can provide input in several ways. One common method is hand raycasting, where a virtual ray is projected from the user’s hand to point at and interact with distant virtual objects or user interface elements. A user can then perform a gesture, such as a pinch with their thumb and index finger, to select a targeted item. For near-field interactions, a user can use direct touch to manipulate virtual objects within arm’s reach. In addition, some systems utilize handheld controllers that the user holds to provide input.SUMMARY

[0004] The described systems and methods provide an ergonomic approach for interacting with an extended reality environment. To reduce the arm fatigue often associated with traditional raycasting, a virtual ray is defined that is aligned with a user’s hand, allowing the arm to remain in a comfortable, lowered position. This alignment is achieved by using the device’s tracking system (i.e., cameras and sensors) to identify the orientation of the user’s hand in three-Atty Docket No. 0120-1137WO1 dimensional space. The virtual ray is then projected outward from the hand, such as in a direction parallel to the user’s palm. For example, the ray can be defined by a vector originating at the user’s wrist and pointing through a knuckle, effectively turning the hand itself into an intuitive pointer. An indicator, such as a cursor, can be displayed on a wearable device, with its position determined based on the defined ray. For instance, if a user rotates their hand to the right, the system recalculates the ray's vector, causing the indicator to move rightward on the display. This enables the user to aim at different virtual objects by adjusting their hand's orientation.

[0005] The system can modify the ray’s behavior to enhance control and comfort. In some implementations, a gain factor is applied to the user’s wrist and forearm movements, amplifying small physical rotations into larger, more responsive movements of the indicator on the display. In some implementations, different gains are applied to different joints; a higher gain on forearm rotation allows for rapid, sweeping navigation, while a lower gain on wrist movement enables precise, fine-tuned adjustments. Furthermore, in some examples, the ray can be translated, for example, shifted vertically, so that a user can keep their hand resting at their side while the indicator appears at a natural height on the display. This combination of techniques enables a user to interact with virtual content accurately and for prolonged periods without physical strain.

[0006] In some aspects, the techniques described herein relate to a method including: defining a ray aligned along at least a portion of a hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

[0007] In some aspects, the techniques described herein relate to a computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method including: defining a ray aligned along at least a portion of a hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

[0008] In some aspects, the techniques described herein relate to a computing system including: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the at least oneAtty Docket No. 0120-1137WO1 processor to perform a method, the method including: defining a ray aligned along at least a portion of a hand of a user; identifying an angle between a forearm of the user and the ray; identifying a gain associated with the angle; and displaying an indicator on a display of a wearable device based on the gain associated with the angle.

[0009] The accompanying drawings and the description below outline the details of one or more implementations. Other features will be apparent from the description, drawings, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 illustrates a computing environment to manage raycasting in extended reality environments according to an implementation.

[0011] FIG. 2 illustrates a method of managing raycast for an extended reality device according to an implementation.

[0012] FIG. 3 illustrates an operational scenario of managing raycast for an extended reality device according to an implementation.

[0013] FIG. 4 illustrates a method of managing raycast for an extended reality device according to an implementation.

[0014] FIG. 5 illustrates an operational scenario of managing raycast for an extended reality device according to an implementation.

[0015] FIG. 6A illustrates an example of user selection in an extended reality environment according to an implementation.

[0016] FIG. 6B illustrates an example of translation of a ray and an indicator according to an implementation.

[0017] FIG. 7 illustrates a computing system to manage raycasting in extended reality environments according to an implementation.DETAILED DESCRIPTION

[0018] The concepts described herein relate to methods for enabling ergonomic hand raycasting in extended reality (XR) environments. A technical problem with traditional ray casting is that it requires users to hold their arm up and point their palm towards a userAtty Docket No. 0120-1137WO1 interface, which can cause significant arm fatigue and discomfort over extended periods (an issue often called “gorilla arm”). This makes prolonged interaction in XR physically taxing.

[0019] The included technical solution provides a set of raycasting techniques where the user’s hand functions like an “invisible controller,” allowing interaction while the arm remains in a comfortable, restful position, typically lowered and close to the body. Instead of a ray originating from the shoulder and passing through a raised palm, the invention defines a raycast vector using a combination of arm joints, such as the elbow and wrist. By applying different control -di splay (CD) gains to the rotational movements of the forearm and wrist, the system allows for both large, sweeping cursor movements and precise, fine-tuned adjustments with minimal physical effort. For example, a user can keep their elbow comfortably at their side and use small forearm and wrist motions to navigate a virtual display, significantly reducing physical strain while maintaining high precision for selecting virtual objects.

[0020] An extended reality (XR) device, often in the form of a head-mounted display (HMD), generates immersive experiences by blending the physical and virtual worlds. These devices can present digital content to a user through various methods. For instance, optical see- through devices, such as augmented reality glasses, use transparent lenses or waveguides to overlay virtual elements directly onto the user’s view of the real world. In contrast, video see- through devices, common in virtual reality headsets, employ external cameras to capture live video of the user’s surroundings and display it on internal screens, seamlessly integrating computer-generated objects into the video feed.

[0021] To interact with virtual objects, these devices often rely on hand tracking to interpret a user’s movements and gestures as input. A prevalent interaction method is hand raycasting, where the system projects a virtual ray from the user’s hand, allowing them to point at and target user interface elements or distant objects. A user can then perform a selection action, typically by making a pinch gesture with their thumb and index finger. A significant technical problem with this approach is that it often requires the user to hold their arm in a raised position for extended periods, leading to physical strain and fatigue. This issue can make prolonged interaction uncomfortable and physically taxing.

[0022] In some technical solutions, an XR device can use its integrated cameras to perform skeletal tracking of the user’s arm. This process involves identifying a plurality of keypoints or features corresponding to anatomical landmarks on the user’s hand, wrist, andAtty Docket No. 0120-1137WO1 forearm. By monitoring the relative positions and orientations of these features, the device constructs a kinematic model of the user’s limb. This model represents the user's limb as a series of connected rigid segments (e.g., forearm, hand) and joints (e.g., elbow, wrist), allowing the system to calculate angles and rotations between different body parts in real-time. This model allows the system to accurately measure the rotational movements of the forearm around the elbow, as well as the flexion and extension of the wrist relative to the forearm, using these isolated motions as input to control a raycast vector. These features, also referred to as keypoints, are specific, identifiable points on the user’s body, such as the wrist, elbow, metacarpophalangeal (MCP) joints, or fingertips, that the device’s cameras detect and track. The three-dimensional coordinates of these keypoints serve as the vertices of the kinematic model, enabling the system to determine the real-time positions and orientations of the user’s limbs.

[0023] As used herein, the term “features” refers to a plurality of specific, identifiable anatomical points (i.e., keypoints) on the user’s body, such as the wrist, elbow, or MCP joints, that are detected and tracked by the device’s skeletal tracking system. The three-dimensional coordinates of these features, also referred to as keypoints, serve as the vertices for a kinematic model of the user’s limb, which provides the positional and orientational data necessary to define the raycast vector.

[0024] A raycast is a computational technique used to determine which virtual objects a projected line, or ray, intersects within a three-dimensional environment. This projected line is mathematically represented as a raycast vector, which is defined by an origin point and a direction. In the context of hand tracking, the ray cast vector’s origin and direction are determined from the user’s kinematic model keypoints. For example, the origin can be mapped to a joint, such as the elbow, and the direction can be calculated from the orientations of the wrist and forearm.

[0025] In some implementations, the device can be configured to define a ray aligned along at least a portion of a hand of a user. As used herein, a ray is a virtual line in three- dimensional space defined by an origin point and a direction vector. A ray is considered to be “aligned along at least a portion of a hand” when its direction is determined by the orientation of the user’s hand, such as being parallel to the palm. To establish this ray, the system utilizes the skeletal tracking data to identify a set of specific keypoints on the hand, such as the wrist, a central palm point, the middle metacarpophalangeal (MCP) joint, and / or the middle proximalAtty Docket No. 0120-1137WO1 phalanx. These keypoints are selected because they form a structurally stable axis of the hand that corresponds to a user’s natural sense of pointing. The direction of the ray is then defined by the vector created between two of these keypoints. To mitigate instability or jitter caused by hand movements or gestures (like a pinch), the system can be configured to dynamically select the two most stable keypoints from this set to define the vector at any given time. This direction vector, combined with an origin point that can be set at the wrist or elbow, fully defines the raycast vector used for interaction. As used herein, a ray is a virtual line in three-dimensional space defined by an origin point and a direction vector. The system projects this ray into the XR environment to perform raycasting, a process that identifies which virtual objects the ray intersects to enable targeting and selection. In some implementations, a first ray is aligned along at least a portion of a hand of a user, and a second ray can be defined as a raycast for interaction with an extended reality environment based on a modification (e.g., applying a gain, translation providing a parallel movement of the ray of the first ray, etc.) to the first ray.

[0026] As used herein, a modification refers to a computational transformation applied to an initial ray (aligned with the user’s hand) to produce a final ray used for interaction within the extended reality environment. A modification can include applying a control-display (CD) gain, which scales the user’s physical joint rotations to amplify the movement of the virtual indicator, and / or applying a translation, which geometrically shifts the ray’s position while preserving its orientation. These transformations are designed to enhance ergonomic control by allowing a user to navigate a virtual interface with minimal physical strain.

[0027] As an illustrative example of displaying an indicator, in a hand-level pointing configuration, the origin of the raycast vector can be fixed to the three-dimensional coordinates of the user’s wrist keypoint as determined by the skeletal tracking system. The direction of the ray is then calculated as the normalized vector pointing from this wrist keypoint to the keypoint representing the middle MCP joint. This configuration effectively creates a ray that projects straight out from the hand, parallel to the palm’s surface. By anchoring the ray’s origin at the wrist and defining its direction with stable hand keypoints, the user can accurately aim while keeping their elbow in a lowered, restful position, mitigating the arm fatigue associated with traditional shoulder-based raycasting.

[0028] To illustrate this interaction, consider a user sitting comfortably with their arm resting at their side. In their view, a virtual user interface is displayed, containing severalAtty Docket No. 0120-1137WO1 interactive buttons. The system, leveraging the skeletal tracking data, defines the ray as originating from the user's wrist and projecting forward in the direction established by the wrist and MCP keypoints, parallel to the palm. This ray is then cast into the virtual environment. Where the ray intersects the plane of the user interface, the system displays a visual indicator, such as a cursor. To move this cursor from one button to another, the user does not need to lift or point their entire arm. Instead, they can make small, subtle rotational movements with their wrist. This slight change in the hand's orientation modifies the direction of the ray, which in turn moves the indicator across the virtual interface, allowing the user to accurately target and hover over the desired button before making a selection.

[0029] As used herein, an “indicator” is a visual element rendered on the display of the wearable device to provide targeting feedback to the user. The position of the indicator is determined by calculating the intersection point of the defined ray with a virtual surface, such as a user interface plane or a three-dimensional object. The indicator can take various forms, such as a cursor, a dot, or a highlight that conforms to the shape of the targeted object, visually communicating to the user where they are aiming within the extended reality environment.

[0030] In some implementations, the system can use a gain amplifier associated with the movement of the ray associated with the user’s palm. This amplification is achieved by applying an angle-display gain, a type of CD gain that mathematically scales the rotational movements of the user’s joints. The system tracks the angular displacement of the wrist relative to the forearm and the rotation of the forearm around the elbow. These small, measured rotations are then multiplied by a predetermined gain factor. For example, a subtle 1 -degree flexion of the wrist can be amplified to produce a 5-degree change in the ray’s direction, enabling the cursor to traverse a large virtual display with minimal physical effort.

[0031] As used herein, “gain” or “control-display (CD) gain” refers to a predetermined scaling factor that defines the mathematical ratio between the movement of a physical control (e.g., the angular displacement of a user's joint) and the corresponding movement of a virtual indicator on a display. An angle-display gain is a specific application of this principle, where the input is a rotational movement and the output is a corresponding angular change in the raycast vector’s direction. By setting a gain factor greater than one, the system can map a small, comfortable physical rotation of a user’s joint into a much larger and more rapid movement of the virtual indicator. The system can apply different gain factors to different joints. For instance,Atty Docket No. 0120-1137WO1 a high gain might be applied to forearm rotation for quick, coarse navigation across the display, while a lower gain is applied to wrist flexion for precise, fine-grained adjustments when targeting a small icon.

[0032] To illustrate, consider a user wearing an XR headset and sitting with their arm in a relaxed position, with their elbow resting comfortably at their side. On a virtual display in front of them, a grid of application icons is shown, and a indicator is currently highlighting a first icon. To move the indicator to a second icon located to the right, the user does not need to lift their arm. Instead, they perform a small, subtle rotation of their forearm and a slight flexion of their wrist. The XR device’s tracking system interprets these minimal movements, using the defined wrist-to-MCP raycast vector to translate them into a larger, corresponding movement of the indicator across the virtual display, smoothly traversing from the first icon to the second icon with precision. This allows for comfortable and sustained interaction without inducing arm fatigue.

[0033] In some implementations, the system can employ an arm-amplified raycast that incorporates data about the arm and the wrist to move the indicator. This method functions as a mixed joint gain controller, combining and scaling rotational displacements from both the elbow and the wrist to create a single, cohesive cursor movement. The system tracks the rotation of the forearm and the flexion and extension of the wrist, using these distinct motions as separate inputs. By applying different CD gains to each of these movements, the system can balance the need for both rapid, large-scale navigation and precise, fine-grained adjustments, all while the user’s arm remains in a comfortable, lowered position.

[0034] To implement this, the system applies a larger gain factor to the rotational movement of the forearm. This allows a small and stable rotation of the arm, with the elbow kept comfortably at the user’s side, to produce a large, sweeping displacement of the cursor across the virtual user interface. This is ideal for quickly traversing the display. In parallel, a smaller gain factor is applied to the wrist’s movement. This enables the user to perform fine, “last-mile” adjustments with high precision, using the lower-energy wrist motion to accurately target small icons or user interface elements without overshooting.

[0035] To calculate the indicator’s final position, the system can determine the rotational angles of the forearm and the wrist in both the horizontal and vertical planes, relative to the user’s forward direction or another stable reference. This stable reference is a coordinate frameAtty Docket No. 0120-1137WO1 or vector used as a consistent baseline against which the system measures these angular movements. While derived from the user’s body, it is established relative to a more stationary part, such as the torso or head, rather than the moving limb itself. For instance, the “user's forward direction” can be defined as a vector aligned with the user's torso. Alternatively, the system can create a “virtual elbow” keypoint, a stable anchor point whose position is calculated relative to the user’s head position as tracked by the wearable device. This ensures that the measured rotations of the forearm and wrist are independent of the user’s overall body movement, providing a reliable basis for calculating cursor displacement. These angular displacements are then multiplied by their respective, predetermined gain factors. The resulting scaled values from the forearm and wrist are combined to compute the final planar displacement of the cursor, effectively translating small, ergonomic joint rotations into a full range of responsive and accurate movement on the virtual display.

[0036] In a wrist-only restful raycast implementation, the system can rely primarily on the angular displacement of the hand relative to the forearm. The system’s skeletal tracking identifies keypoints on the hand and forearm to construct a kinematic model, from which it calculates the angle between the forearm’s vector and the hand-aligned ray. This angle directly corresponds to the user’s wrist flexion and extension. A single control-display (CD) gain is then applied to this angle. As used herein, a CD gain is a predetermined scaling factor that defines the ratio between the angular displacement of the virtual indicator and the corresponding angular displacement of the user’s physical joint. By multiplying the measured wrist angle by this gain factor, the system amplifies small, low-effort wrist movements into larger, more responsive movements of the indicator on the display.

[0037] To provide a more nuanced control scheme that balances rapid navigation with high precision, the system can be configured as a mixed joint gain controller. This approach incorporates rotational movements of the wrist and forearm as distinct inputs. In addition to the wrist angle, which is the angle between the forearm’s vector and the hand-aligned ray, the system identifies a second angle representing the rotation of the forearm itself. This second angle is measured relative to a stable reference vector (i.e., another vector associated with another portion of the user), such as a vector aligned with the user's torso or a virtual elbow keypoint derived from the head's position as tracked by the wearable device. A second, typically larger, gain factor is associated with this forearm rotation. The higher gain on the forearm movementAtty Docket No. 0120-1137WO1 allows for large, sweeping cursor displacements with minimal physical motion, ideal for quickly traversing a large display.

[0038] As used herein, the term “another vector associated with another portion of the user” is a stable reference vector against which forearm rotation is measured and can be defined in several ways. In a first definition, it refers to a stable reference vector derived from a portion of the user’s body that is less mobile than the user’s arm, such as the torso or head, which serves as a consistent baseline for measuring the rotational movement of the forearm independent of the user's overall body movement. In a second definition, it refers to a vector defined within a coordinate system anchored to the user's torso or head, established to provide a stationary directional reference against which the angular displacement of the user's forearm is calculated. In a third definition, it is a vector calculated by the wearable device's tracking system to represent a consistent directional baseline, determined from skeletal tracking keypoints associated with the user’s torso or head, thereby creating a reference that remains stable relative to the user's body orientation.

[0039] The final position of the indicator is then computed by combining the scaled outputs from both joints. The system can measure the angular displacements of the wrist and forearm in both the horizontal and vertical planes. Each measured angle is multiplied by its respective gain factor — the lower gain for the wrist and the higher gain for the forearm. The resulting scaled values are then summed to calculate the final planar displacement of the indicator on the user interface. This mathematical combination allows the lower-energy wrist motion to be used for fine, “last-mile” adjustments, while the more stable forearm rotation handles coarse, high-speed navigation. The technical effect is an interaction model that is both ergonomically efficient and highly precise, allowing a user to maintain their arm in a restful position while achieving a full range of control.

[0040] For example, consider a user sitting comfortably on a couch with their arm resting at their side. In front of them is a large virtual user interface displaying a web browser. To move the cursor from its current position on the left side of the screen to a link on the far right, the user performs a small, subtle rotation of their forearm. The system, applying a high control-display gain to this forearm movement, translates the small physical rotation into a large, sweeping motion of the cursor across the display. This allows the user to traverse the wide virtual spaceAtty Docket No. 0120-1137WO1 quickly and with minimal physical effort, without needing to lift their arm from its restful position.

[0041] Once the cursor is in the general vicinity of the target link, the user needs to make a precise adjustment. To do this, they use a slight flexion or extension of their wrist. A lower control -di splay gain is applied to this wrist motion, which translates the small hand movement into a much slower, more controlled movement of the cursor on the screen. This low-gain, high- precision control allows the user to easily perform the “last-mile” adjustment to accurately position the cursor directly over the small hyperlink without overshooting it. By combining high- gain forearm movement for speed and low-gain wrist movement for precision, the system enables an interaction that is both fast and accurate while remaining ergonomically comfortable for prolonged use.

[0042] In some implementations, the system first defines a ray aligned along at least a portion of a user’s hand. This is achieved by using the wearable device’s tracking system to perform skeletal tracking, identifying the three-dimensional coordinates of anatomical keypoints on the hand, such as the wrist and the MCP joint. These keypoints are used to define a vector that is parallel to the user’s palm, establishing an intuitive pointing direction. The system then determines a modification to this ray based on a relationship between the ray and another portion of the user, such as the forearm. The final position of an indicator on the device’s display is then calculated and rendered based on this determined modification, allowing for ergonomic control.

[0043] As used herein, the “relationship between the ray and the portion of the user” is a kinematic property derived from a skeletal tracking model of the user. This property can be a spatial orientation defined by a measured angular displacement between a vector representing the ray and a vector representing the portion of the user. It can also be a geometric property calculated between the ray and the portion of the user in a three-dimensional coordinate space, including an angle, a distance, or a rate of change thereof. This calculated relationship serves as a control input for determining the modification. In some implementations, this relationship can be quantified as the angle between the hand-aligned ray and the user’s forearm, which directly measures wrist flexion and extension. The modification, in this case, is a CD gain, a predetermined scaling factor, applied to this angle to amplify small wrist movements into larger, more responsive movements of the indicator.Atty Docket No. 0120-1137WO1

[0044] To calculate the angle between the ray and the forearm, the system can first define the forearm as a vector originating at the user’s elbow keypoint and terminating at the user’s wrist keypoint, as determined by the skeletal tracking model. The angle is then computed between this forearm vector and the hand-aligned ray vector. To provide a more nuanced control scheme that leverages both coarse and fine motor control, the system can function as a mixed joint gain controller. In this implementation, the modification is determined by also identifying a second relationship: a second angle representing the rotational movement of the forearm itself, measured relative to a stable reference vector (e.g., a vector aligned with the user’s torso). A second, typically larger, gain is associated with this forearm rotation, allowing the higher gain on the stable forearm to facilitate rapid, coarse navigation, while the lower gain on the wrist enables fine-grained, high-precision adjustments. In another implementation, the modification can be a translation, which geometrically shifts the ray to a new position while preserving its orientation, creating a parallel ray. This allows a user to keep their hand in a low, restful position while the indicator appears at a more natural height on the display.

[0045] The technical effects of these innovations culminate in a hand-raycasting system that is significantly more ergonomic, precise, and reliable than traditional methods. By defining the raycast vector from joints like the elbow and wrist and applying control-display gains, the system eliminates the need for users to hold their arms in fatiguing raised positions, directly mitigating the “gorilla arm” problem. The implementation of a mixed joint gain controller creates a highly efficient interaction model where high gain on forearm rotation facilitates rapid, sweeping cursor movements across large displays, while lower gain on wrist flexion enables fine-grained, high-accuracy adjustments for targeting small elements. Furthermore, by dynamically defining the ray from the most stable anatomical keypoints on the hand, the system mitigates instability and jitter, ensuring consistent targeting during interaction. Ultimately, these techniques transform the user’s hand into a high-fidelity “invisible controller,” enabling comfortable, intuitive, and sustained engagement within XR environments without the encumbrance of a physical device.

[0046] FIG. 1 illustrates a computing environment 100 to manage ray casting in extended reality environments according to an implementation. Computing environment 100 includes user 110, XR device 130, and user view 141. XR device 130 includes display 131, sensors 132, camera 133, and ray application 126. User view 141 further includes interface 143, indicator 144,Atty Docket No. 0120-1137WO1 and hand 145. Ray application 126 can be stored in a non -transitory storage medium and executed across one or more processors. In some implementations, at least a portion of ray application 126 can be implemented on a companion computing device to XR device 130 such as, a smartphone, a tablet, a laptop, and / or so forth.

[0047] In computing environment 100, user 110 wears XR device 130, which executes ray application 126. Ray application 126 utilizes sensors 132 and camera 133 to track the arm and hand 145 of user 110. Based on this tracking data, the application defines a virtual ray that controls the position of indicator 144 on interface 143, as viewed by user 110 on display 131. This system allows user 110 to navigate the interface by making small, comfortable movements of their wrist and forearm, enabling precise interaction without requiring the user to hold their arm in a raised, fatiguing position.

[0048] Ray application 126 defines this ray by first performing skeletal tracking on hand 145 using data from camera 133 and / or sensors 132. This process identifies a plurality of anatomical features, or keypoints, on the user’s hand in three-dimensional space. These keypoints can include the wrist, a central palm point, the MCP joint, or the middle proximal phalanx. The system can then define a ray that is parallel to the palm of hand 145. The origin of this ray can be set to the keypoint corresponding to the user’s wrist, while its direction is determined by the vector created between two of the other keypoints, such as the vector from the wrist to the middle MCP joint. To ensure stability, the system can dynamically select the two most stable keypoints to define this direction vector, mitigating jitter from hand movements.

[0049] As used herein, a “ray” is a virtual line in three-dimensional space, mathematically defined by an origin point and a direction vector. In this system, the ray is projected from the user’s hand along the line of the user’s palm. The “features” of the hand refer to the specific anatomical keypoints identified by the skeletal tracking system. These include, but are not limited to, the wrist, various joints on the hand such as the MCP joints, and fingertips. The three-dimensional coordinates of these features are used to construct a kinematic model of the hand, providing the data necessary to define the ray’s origin and direction.

[0050] Once the ray is defined in association with hand 145, the device can determine a location for indicator 144. In some implementations, the location for indicator 144 corresponds to the ray associated with hand 145. To do so, the system projects this ray into the virtual environment and calculates the point at which the ray intersects with interface 143. Indicator 144Atty Docket No. 0120-1137WO1 is then displayed at this intersection point, providing user 110 with visual feedback for targeting and selection.

[0051] In some implementations, one or more modifications are applied to the ray associated with hand 145 to manage the translation of the location of indicator 144 on interface 143. These modifications can include applying a gain, providing a translation or a parallel movement of the ray associated with the user’s hand, or a combination of the two. Referring first to gain, the system can apply a CD gain, which is a predetermined scaling factor that defines the ratio between the movement of a physical control and the corresponding movement of indicator 144 on interface 143. In this context, the system tracks the angular displacement of the user’s joints, such as the rotation of the wrist relative to the forearm. This small, measured physical rotation is then multiplied by the gain factor, which amplifies the movement. The result is that a small, comfortable rotation of the user’s wrist or forearm is translated into a much larger and more rapid movement of the ray, and consequently, a larger displacement of indicator 144 on the display. This allows user 110 to navigate across a wide virtual interface with minimal physical effort, reducing arm fatigue.

[0052] Referring next to the parallel translation of the determined ray, this modification involves shifting the ray to a new position while preserving its original orientation, resulting in a second ray that is parallel to the first. The location of indicator 144 is then based on this second, translated ray. This technique provides a significant ergonomic benefit by allowing user 110 to keep their hand 145 in a low, restful position — for example, at their side or on their lap — while indicator 144 appears at a more natural height on interface 143. Without translation, a ray originating from a lowered hand would point towards the floor or the bottom of the interface. By applying a vertical translation, the system remaps this low-pointing ray to the vertical center of the user interface, decoupling the physical height of the user’s hand from the vertical position of the indicator and allowing interaction without arm fatigue. It should be understood that a system can employ both gain and parallel translation in some examples.

[0053] In some implementations, ray application 126 can be configured to as a mixed joint gain controller that uses both the motion of the wrist and the motion of the forearm to support the movement of an indicator in the environment. This technique functions by combining scaled rotational displacements from both the elbow and the wrist to produce a single, cohesive movement of indicator 144. The system tracks the rotation of the user’s forearm and the flexionAtty Docket No. 0120-1137WO1 of their wrist as two distinct inputs. By applying different CD gains — predetermined scaling factors — to each of these movements, the application can balance the need for both rapid, large- scale navigation and precise, fine-grained adjustments, all while user 110 keeps their arm in a comfortable, lowered position.

[0054] Specifically, a larger gain factor is applied to the rotational movement of the forearm. This allows a small and stable rotation of the arm, with the elbow kept comfortably at the user’s side, to produce a large, sweeping displacement of indicator 144 across interface 143. This high-gain component is ideal for quickly traversing the display with minimal physical effort.

[0055] In parallel, a smaller gain factor is applied to the wrist’s movement. This enables user 110 to perform fine, “last-mile” adjustments with high precision, using the lower-energy wrist motion to accurately target small icons or user interface elements without overshooting. To calculate the final position of indicator 144, the system determines the rotational angles of the forearm and the wrist, multiplies these angular displacements by their respective gain factors, and combines the resulting scaled values to compute the final planar displacement of the indicator on the display. This effectively translates small, ergonomic joint rotations into a full range of responsive and accurate movement.

[0056] FIG. 2 illustrates method 200 of managing raycast for an extended reality device according to an implementation. Method 200 may be implemented within a system that includes an Extended Reality (XR) device, such as XR device 130. In some implementations, method 200 can be implemented by computing system 700 of FIG. 7.

[0057] Method 200 includes defining a ray aligned along at least a portion of a hand of a user at step 201. This step can be performed by the XR device using its integrated sensors, such as cameras, to capture one or more images of the user’s hand and arm. From these images, the system performs skeletal tracking to identify the three-dimensional coordinates of a plurality of anatomical features or keypoints. These keypoints correspond to specific features of the hand, such as the wrist, a central point on the palm, the metacarpophalangeal (MCP) joints, and the proximal phalanges. By monitoring the positions and orientations of these keypoints, the device constructs a real-time kinematic model of the user’s hand, which provides the foundational data for determining the ray’s origin and direction.Atty Docket No. 0120-1137WO1

[0058] Based on the data from the kinematic model, the system defines the ray as a vector in three-dimensional space. The origin of this vector can be fixed to a specific keypoint, such as the user’s wrist, allowing the user’s arm to remain in a comfortable, lowered position. The direction of the vector is then calculated to be parallel to the surface of the user’s palm. This is accomplished by defining the vector between two selected features or keypoints on the hand, such as the vector pointing from the wrist keypoint to the middle MCP joint keypoint. This configuration creates a ray that projects forward from the hand, mimicking the intuitive pointing action of a physical controller.

[0059] To ensure the ray remains stable and precise, particularly when the user is performing a gesture such as a pinch to select an item, the system can be configured to dynamically select the most stable keypoints available to define the ray’s direction. For instance, the system may have a set of four potential features or keypoints (e.g., wrist, palm center, middle MCP, and middle proximal phalanx) and can choose the two keypoints that exhibit the least movement or jitter at any given moment. This adaptive approach mitigates instability caused by hand movements, ensuring that the raycast interaction is reliable and accurate.

[0060] Method 200 further includes displaying an indicator on a display of a wearable device based on the ray at step 202. Once the ray is defined, the system projects this ray into the extended reality environment to determine its intersection point with virtual content. This process, known as ray casting, calculates where the virtual line intersects with user interface planes or three-dimensional objects. At this point of intersection, the XR device renders a visual indicator on the user’s display. This indicator, which can take the form of a cursor, a dot, or a highlight that conforms to the shape of a targeted object, provides the user with immediate visual feedback, showing them precisely where they are aiming within the virtual scene.

[0061] The movement or location of this indicator can be based on a modification of the initial ray defined in step 201. For example, the system can apply a control -di splay (CD) gain to amplify the user’s physical movements. This means that a small rotational movement of the user’s wrist or forearm is scaled by a predetermined gain factor, resulting in a larger corresponding movement of the indicator on the display. For instance, in a wrist-only configuration, the system can apply a gain factor to the angle between the user’s forearm and their hand. A user looking at a virtual display might only need to flex their wrist a few degrees to the left or right. The system multiplies this small angular change by the gain factor, causing theAtty Docket No. 0120-1137WO1 indicator to sweep across a much larger arc on the display, enabling rapid navigation with minimal physical effort. As used herein, “gain” refers to the control-display gain, which is a predetermined scaling factor that defines the ratio between the angular displacement of the virtual indicator and the corresponding angular displacement of the user’s physical joint. A gain factor greater than one amplifies the user’s motion, allowing a small rotation of the wrist or forearm to produce a much larger and more rapid movement of the indicator on the display. In a mixed joint gain controller implementation, the system can apply a higher gain to forearm rotation for coarse, high-speed navigation and a lower gain to wrist flexion for fine-grained, high-precision adjustments. This allows the user to comfortably and accurately control the indicator across a large virtual space while keeping their arm in a restful position.

[0062] In some implementations, the system can employ a translation associated with the ray defined in step 201. This modification involves shifting the entire ray to a new position in space while preserving its original orientation, resulting in a second ray that is parallel to the first. The indicator on the display is then based on this second, translated ray. For example, the system might define an initial ray aligned with the user’s palm but then apply a vertical translation to shift this ray upwards. The final indicator shown on the display would therefore appear higher than where the user’s hand is actually pointing, while still moving left and right in direct correspondence with the hand’s orientation. As used herein, a translation is a geometric transformation that displaces a ray from its original position to a new position, such that the resulting ray is parallel to the original ray. This technique provides a significant ergonomic benefit by allowing the user to keep their hand in a low, restful position, such as at their side or on their lap, while the indicator appears at a more natural height within their field of view. For instance, without translation, a ray originating from a lowered hand would naturally point towards the floor or the bottom edge of a virtual display. By applying a vertical translation, the system can remap this low-pointing ray to the vertical center of the user interface, effectively decoupling the physical height of the user’s hand from the vertical position of the indicator. The technical effect of this is a reduction in physical strain, as the user can interact with content at eye-level without needing to physically lift their arm into a fatiguing position.

[0063] In some implementations, a device can use multiple joints or portions of the user’s body to define the movement of the indicator for the wearable device. This can be implemented using a mixed joint gain controller, a technique where the system combines scaled rotationalAtty Docket No. 0120-1137WO1 displacements from both the elbow and the wrist to produce a single, cohesive movement of the indicator. The system tracks the rotation of the user’s forearm around the elbow and the flexion and extension of the wrist as two separate inputs. By applying different, predetermined controldisplay gains to each of these movements, the device can balance the need for both rapid, large- scale navigation and precise, fine-grained adjustments, all while the user’s arm remains in a comfortable, lowered position.

[0064] Specifically, a larger gain factor is applied to the rotational movement of the forearm. This allows a small and stable rotation of the arm, with the elbow kept comfortably at the user’s side, to produce a large, sweeping displacement of the indicator across the virtual user interface. This high-gain component is ideal for quickly traversing the display with minimal physical effort. In parallel, a smaller gain factor is applied to the wrist’s movement. This enables the user to perform fine, “last-mile” adjustments with high precision, using the lower-energy wrist motion to accurately target small icons or user interface elements without overshooting.

[0065] To calculate the indicator’s final position, the system determines the rotational angles of the forearm and the wrist in both the horizontal and vertical planes, relative to a stable reference such as the user’s forward direction. These angular displacements are then multiplied by their respective gain factors. The resulting scaled values from the forearm and wrist are combined to compute the final planar displacement of the indicator, effectively translating small, ergonomic joint rotations into a full range of responsive and accurate movement on the virtual display. The technical effect of this mixed joint controller is a significant improvement in both ergonomic comfort and interaction efficiency. By allowing the user to keep their arm in a restful, lowered position, the system directly mitigates the physical strain and fatigue associated with traditional raycasting methods that require a raised arm. The dual -gain approach creates a highly effective interaction model: the high gain applied to forearm rotation enables rapid, sweeping movements of the indicator across large virtual displays with minimal physical effort, while the lower gain on wrist flexion provides the fine-grained control necessary for precise, “last-mile” adjustments to accurately target small user interface elements. This combination of high-speed navigation and high-precision targeting transforms the user’s hand into a versatile and reliable input device, enabling comfortable and sustained interaction within the extended reality environment.Atty Docket No. 0120-1137WO1

[0066] To support the mixed joint gain controller, the system first identifies the angle between the user’s forearm and the hand-aligned ray. This angle is measured by constructing a kinematic model from the skeletal tracking data. The forearm is represented as a first vector, for instance, a vector defined by the three-dimensional coordinates of the elbow and wrist keypoints. The hand-aligned ray, which is defined by keypoints on the hand (e.g., from the wrist to the MCP joint), serves as a second vector. The angle between these two vectors in three-dimensional space provides a quantitative measure of the user’s wrist flexion and extension. A first CD gain is then associated with this angle. This gain is a predetermined scaling factor that amplifies the measured angular displacement, allowing small, low-effort wrist movements to be translated into larger, more responsive movements of the indicator on the display.

[0067] In addition, the system identifies a second angle to facilitate large-scale navigation. This second angle represents the rotational movement of the forearm itself, measured relative to a stable reference vector associated with another portion of the user’s body. This reference can be a “torso forward vector,” aligned with the user’s torso, or a vector derived from a “virtual elbow” feature, whose position is calculated relative to the user’s head as tracked by the wearable device. This ensures that forearm rotation is measured independently of the user's overall body movement. A second, typically larger, gain is associated with this forearm rotation. The final position of the indicator is then computed by combining the scaled outputs from both joints. The system continuously measures the angular displacements of both the wrist and forearm, multiplies each by its respective gain factor, and sums the results to calculate the final planar displacement of the indicator.

[0068] For example, to move the indicator from the left side of a large virtual display to a small icon on the far right, a user first performs a small rotation of their forearm while keeping their elbow at their side. The system applies the high gain factor associated with the forearm's angle, causing the indicator to sweep rapidly across the display to the general vicinity of the icon. Once close, the user makes a slight flexion of their wrist. The system applies the lower gain factor based on the angle between the forearm and the ray, translating this small wrist movement into a slow, controlled motion of the indicator. This allows the user to easily perform the “last- mile” adjustment to accurately position the indicator directly over the icon before making a selection.Atty Docket No. 0120-1137WO1

[0069] In at least one example operation of a wearable device, the system first defines a ray aligned along at least a portion of a user’s hand. This is achieved by using the wearable device’s tracking system to perform skeletal tracking, identifying the three-dimensional coordinates of anatomical keypoints on the hand, such as the wrist and the MCP joint. These keypoints are used to define a vector that is parallel to the user’s palm, establishing an intuitive pointing direction. The system then determines a modification to this ray based on a relationship between the ray and another portion of the user, such as the forearm. The final position of an indicator on the device’s display is then calculated and rendered based on this determined modification, allowing for ergonomic control.

[0070] In some operations, the relationship is quantified as the angle between the hand- aligned ray and the user’s forearm, which directly measures wrist flexion and extension. The modification, in this case, is a control-display gain, a predetermined scaling factor, applied to this angle to amplify small wrist movements into larger, more responsive movements of the indicator. To provide a more nuanced control scheme, the system can function as a mixed joint gain controller by also identifying a second angle representing the rotational movement of the forearm itself, measured relative to another stable vector associated with another portion of the user. A second, typically larger, gain is associated with this forearm rotation, allowing the higher gain on the stable forearm to facilitate rapid, coarse navigation, while the lower gain on the wrist enables fine-grained, high-precision adjustments. Alternatively, the modification can be a translation, which geometrically shifts the ray to a new position while preserving its orientation, creating a parallel ray. This allows a user to keep their hand in a low, restful position while the indicator appears at a more natural height on the display.

[0071] FIG. 3 illustrates an operational scenario 300 of managing ray cast for an extended reality device according to an implementation. Operational scenario 300 includes hand 305 ray 307, wrist movement 310, interface 315, and indicator movement 320.

[0072] In operational scenario 300, an XR device monitors hand 305 and its corresponding wrist movement 310 using integrated cameras and sensors to perform skeletal tracking. This process identifies the three-dimensional coordinates of a plurality of anatomical features, or keypoints, on the user’s hand, such as the wrist, a central palm point, and the MCP joints. By tracking the relative positions of these keypoints, the system constructs a real-time kinematic model that accurately measures the orientation of hand 305 and the angularAtty Docket No. 0120-1137WO1 displacement associated with wrist movement 310, such as flexion and extension relative to the forearm.

[0073] Based on this tracking data, the system defines ray 307 as a virtual vector that is aligned along a portion of hand 305, typically configured to be parallel to the palm. The origin of ray 307 is fixed to a stable keypoint, such as the user’s wrist, while its direction is determined by the vector created between two selected keypoints on the hand. For instance, the direction can be calculated as the vector pointing from the wrist keypoint to the middle MCP joint keypoint. This configuration establishes a stable and intuitive pointing direction that projects forward from the hand while the user’s arm remains in a comfortable, restful position.

[0074] The device then translates wrist movement 310 into indicator movement 320 on interface 315 by applying a gain multiplier. The system applies a predetermined CD gain to the small angular changes detected from wrist movement 310. This gain factor mathematically scales the physical rotation, so a subtle flexion or extension of the wrist results in a much larger and more rapid change in the direction of ray 307. When this amplified ray is projected onto interface 315, it produces the large-scale indicator movement 320, allowing the user to navigate across the entire interface with minimal physical effort and improved ergonomic comfort.

[0075] In some implementations, the indicator displayed by the device can be translated. This translation is a geometric transformation that displaces the ray from its original position to a new one while preserving its orientation, resulting in a second ray that is parallel to the first. The location of the indicator on interface 315 is then based on this second, translated ray. This technique provides a significant ergonomic benefit by decoupling the physical height of hand 305 from the vertical position of the indicator. For example, if a user’s hand is in a low, restful position at their side or on their lap, the initial ray 307 would naturally point toward the bottom of interface 315 or even the floor. By applying a vertical translation, the system can remap this low-pointing ray to a more natural, eye-level height on the interface. This allows the user to interact with the full vertical range of the interface without needing to physically lift their arm into a fatiguing position, directly addressing the physical strain associated with traditional raycasting methods.

[0076] FIG. 4 illustrates a method 400 of managing ray cast for an extended reality device according to an implementation. The steps of method 400 can be performed by an XR device,Atty Docket No. 0120-1137WO1 such as XR device 130 in FIG. 1 . Tn some implementations, method 400 may be implemented by computing system 700 of FIG. 7.

[0077] Method 400 includes identifying a first movement of a hand relative to a forearm and identifying a second movement of the forearm relative to another portion of the body at step 401. This step is performed by the XR device using its integrated cameras and sensors to conduct skeletal tracking of the user’s arm. The system identifies a plurality of anatomical features, such as the wrist, elbow, and points along the forearm, in three-dimensional space. By determining the relative positions and orientations of these keypoints, the device can construct a kinematic model of the user’s limb. The “first movement” is then identified as the angular displacement of the hand relative to the forearm, which corresponds to wrist flexion and extension. The “second movement” is identified as the rotational displacement of the forearm around the elbow, measured relative to a stable reference point, such as a virtual elbow keypoint derived from the user’s head position or a vector representing the user’s torso.

[0078] Method 400 further includes identifying a first gain associated with the first movement and a second gain associated with the second movement at step 402, and displaying an indicator based on the first gain and movement and the second gain and movement at step 403. In step 402, the system identifies two distinct CD gains, which are predetermined scaling factors that define the ratio between a physical joint’s rotation and the corresponding movement of the indicator on the display. A first gain is associated with the first movement (wrist flexion / extension), and a second, typically larger, gain is associated with the second movement (forearm rotation). The use of two different gains is a core component of the mixed joint gain controller strategy. The second gain, applied to the more stable forearm, is set to a higher value to enable large, sweeping displacements of the indicator with minimal physical effort. This allows for rapid traversal of a large virtual interface. The first gain, applied to the wrist, is set to a lower value, allowing the user to perform fine, high-precision adjustments for targeting small elements without overshooting.

[0079] Step 403 involves using these gains and movements to calculate the indicator’s position and render it on the display. The system continuously calculates the final planar displacement of the indicator by combining the outputs from both joints. To do this, it measures the angular displacements of the wrist and the forearm in both the horizontal and vertical planes, relative to a stable reference like the user’s forward direction. Each of these angular values isAtty Docket No. 0120-1137WO1 then multiplied by its corresponding gain factor (the first gain for wrist movement, the second gain for forearm movement).

[0080] The resulting scaled values from both the wrist and forearm are then summed to compute the final two-dimensional coordinates for the indicator on the user interface. This process effectively translates two separate, small-scale physical rotations into a single, cohesive, and highly responsive movement of the indicator on the virtual display. By mathematically combining these inputs, the system creates a seamless control experience that leverages the strengths of each joint’s range of motion.

[0081] The technical effect of this process is a highly ergonomic and efficient interaction model. The high gain applied to the forearm rotation facilitates rapid, coarse navigation across the display, allowing the user to quickly move the indicator to the general vicinity of a target with a small, comfortable arm rotation. Subsequently, the lower gain on the wrist movement enables fine-grained, “last-mile” adjustments, providing the precision needed to accurately select small icons or hyperlinks. This dual-gain approach mitigates the physical strain of traditional ray casting by allowing the user’s arm to remain in a restful position, transforming the hand into a versatile input device that is both fast for navigation and accurate for selection.

[0082] In some implementations, the system can first identify an angle between the user’s forearm and the hand-aligned ray to quantify the first movement. This angle is measured from the kinematic model constructed via skeletal tracking. The forearm is represented as a first vector (e.g., defined by the elbow and wrist keypoints), and the hand-aligned ray is represented as a second vector (e.g., defined by the wrist and MCP keypoints). The angle between these two vectors provides a direct measurement of the user’s wrist flexion and extension, and the first gain is associated with this angle.

[0083] In addition, the system identifies a second angle to quantify the second movement, which is the rotational displacement of the forearm itself. This second angle is measured relative to a stable reference vector associated with another portion of the user, such as a vector aligned with the user’s torso or a virtual elbow keypoint whose position is derived from the head’s position as tracked by the wearable device. This ensures forearm rotation is measured independently of the user’s overall body movement. The second gain is associated with this second angle. The final position of the indicator is then computed by combining the scaled outputs from both joints; the angular displacements of both the wrist and forearm are multipliedAtty Docket No. 0120-1137WO1 by their respective gains and then summed to calculate the final planar displacement of the indicator.

[0084] FIG. 5 illustrates an operational scenario 500 of managing raycast for an extended reality device according to an implementation. Operational scenario 500 includes top view 510 and side view 511 of a user arm. Top view 510 further includes wrist angle 530 and forearm angle 531. Side view 511 further includes wrist angle 532 and forearm angle 533. Angles 530- 533 are used to determine movement (delta X 540 and delta Y 541). Operational scenario 500 demonstrates the movement of an indicator 550 on interface 520 based on the movement of the user’s arm.

[0085] In operational scenario 500, a wearable device can use integrated cameras and sensors to perform skeletal tracking of the user’s arm in three-dimensional space. This process identifies the positions of anatomical keypoints, such as the elbow, wrist, and points along the forearm, to construct a real-time kinematic model of the limb. From this model, the device calculates the rotational displacements of the joints. Wrist angle 530 and forearm angle 531 represent the angular movements in the horizontal plane, corresponding to left -right motion. Wrist angle 532 and forearm angle 533 represent the angular movements in the vertical plane, corresponding to up-down motion. The forearm angles 531 and 533 are measured as the rotation of the forearm around the elbow relative to a stable reference, such as the user’s forward direction, while the wrist angles 530 and 532 are measured as the flexion and extension of the hand relative to the forearm.

[0086] These four angular measurements are then used as inputs to calculate the final displacement of the indicator on interface 520, shown as delta X 540 (horizontal movement) and delta Y 541 (vertical movement). The system applies separate, predetermined control -di splay gains to the wrist and forearm movements. The rotational values from the forearm angles 531 and 533 are typically multiplied by a higher gain factor to enable rapid, large-scale traversal of the interface with small arm movements. In parallel, the rotational values from the wrist angles 530 and 532 are multiplied by a lower gain factor, allowing for high-precision, “last-mile” adjustments. The scaled outputs from the forearm and wrist are then combined to compute the final planar movement of the indicator, translating small, ergonomic joint rotations into a full range of responsive and accurate cursor control.Atty Docket No. 0120-1137WO1

[0087] In some implementations, to determine the movement of the cursor from a first position to a second position, the device determines the deltas or changes in the angles 530, 531, 532, and 533 from a first time to a second time. For example, wrist angle 530 may have been approximately +20 degrees at a first time and -20 degrees at a second time; forearm angle 531 may have been approximately 20 degrees at a first time and -10 degrees at a second time; wrist angle 532 may have been approximately +20 degrees at a first time and +20 degrees at a second time; and forearm angle 533 may have been approximately 20 degrees at a first time and 0 degrees at a second time. The delta of each of these angles 530-533 can then be used to calculate the final cursor position on interface 520, which is shown as delta X 540 and delta Y 541.

[0088] For example, to move indicator 550 toward a target button located on interface 520, the user can perform a small rotation of their forearm. The system applies a high controldisplay gain to forearm angles 531 and 533, translating this subtle motion into a large, rapid movement that quickly brings indicator 550 into the general vicinity of the button. To perform the final, precise adjustment, the user then uses a slight flexion of their wrist. The system applies a lower gain to wrist angles 530 and 532, resulting in a much slower, more controlled movement. This allows the user to accurately position indicator 550 directly over the target button without overshooting, preparing for a selection action.

[0089] FIG. 6A illustrates an example 600 of user selection in an extended reality environment according to an implementation. Example 600 includes hand 608, tap selection 610, interface 615, and indicator 620.

[0090] In example 600, after the user positions indicator 620 over a desired element on interface 615 using the ergonomic ray casting methods described previously, the user can perform a selection action. This selection can be accomplished using the thumb as a selector, as illustrated by tap selection 610. The user performs a simple, intuitive gesture, such as tapping their thumb against their index finger, to confirm a selection without changing hand position or disrupting the raycast.

[0091] The device detects this tap selection by using its skeletal tracking capabilities to monitor the positions of specific anatomical keypoints on the user’s hand 608. The system tracks the three-dimensional coordinates of features on both the thumb and the index finger, such as the thumb tip, the index fingertip, and the index distal and intermediate joints. A “tap” input is registered when the system determines that the keypoint corresponding to the user’s thumb tipAtty Docket No. 0120-1137WO1 makes contact with, or comes within a predefined proximity threshold of, a keypoint on the index finger.

[0092] Upon detecting this gesture, the system interprets the input as a selection command, which then activates the virtual element currently targeted by indicator 620 on interface 615. This method provides a reliable, low-effort mechanism for selection that complements the restful raycasting control, allowing for a complete interaction cycle (point and select) while maintaining ergonomic comfort.

[0093] For example, a user with their arm resting at their side may wish to select a button on a virtual interface. They first perform a small rotation of their forearm, which, due to a high control -di splay gain, causes indicator 620 to sweep rapidly across the screen into the general vicinity of the target button. Next, they use a slight flexion of their wrist, which, with a lower gain, allows for a fine-grained adjustment to precisely position the indicator over the button. Once the button is targeted, the user performs the tap selection 610 gesture by touching their thumb to their index finger. The system detects this contact, interprets it as a selection command, and activates the button.

[0094] FIG. 6B illustrates an example 650 of translation of a ray and an indicator according to an implementation. Example 650 includes hand 658, parallel translation 660, interface 665, and indicator 670.

[0095] In example 650, the system employs a parallel translation 660 to enhance the ergonomic comfort of the interaction. The primary purpose of this technique is to decouple the physical height of the user’s hand 658 from the vertical position of the indicator 670 on the interface 665. This allows the user to keep their arm in a low, restful position — for example, resting at their side or on their lap — while the indicator appears at a more natural, eye-level height on the display. By eliminating the need for the user to physically lift their arm to point at the interface, this method directly mitigates the physical strain and fatigue associated with traditional raycasting.

[0096] The parallel translation works by applying a geometric transformation to the ray that is initially defined by the orientation of the user’s hand. As used herein, a translation is a transformation that shifts a ray from its original position to a new position in space while preserving its orientation, resulting in a second ray that is parallel to the first. The system first defines an initial ray aligned with the user’s palm. It then applies a predetermined offset,Atty Docket No. 0120-1137WO1 illustrated by parallel translation 660, to shift this ray. The final position of indicator 670 is then determined by calculating where this second, translated ray intersects with interface 665.

[0097] For instance, if a user is sitting with their hand 658 resting comfortably on their lap, a ray originating directly from their hand would point towards the floor or the bottom edge of interface 665. By applying a vertical translation, the system remaps this low-pointing ray upwards, allowing indicator 670 to appear in the center of the user’s view. The indicator’s horizontal movement still corresponds directly to the left-right orientation of the user’s hand, but its vertical position is independent of the hand’s physical height, enabling comfortable and sustained interaction without arm fatigue.

[0098] FIG. 7 illustrates a computing system 700 to manage raycasting in extended reality environments according to an implementation. Computing system 700 is representative of any computing system or systems with which the various operational architectures, processes, scenarios, and sequences disclosed herein can be implemented to provide raycasting and displaying indicators as described herein. Computing system 700 may represent a wearable computing device, such as an XR device or smart glasses. Computing system 700 can include multiple computing devices in some examples (e.g., a wearable device and a companion device, such as a smartphone or tablet). Computing system 700 includes storage system 745, processing system 750, communication interface 760, and input / output (VO) device(s) 770. Processing system 750 is operatively linked to communication interface 760, VO device(s) 770, and storage system 745. In some implementations, communication interface 760 and / or VO device(s) 770 may be communicatively linked to storage system 745. Computing system 700 may further include other components, such as a battery and enclosure, that are not shown for clarity.

[0099] Communication interface 760 comprises components that communicate over communication links, such as network cards, ports, radio frequency, processing circuitry and software, or some other communication devices. Communication interface 760 may be configured to communicate over metallic, wireless, or optical links. Communication interface 760 may be configured to use Time Division Multiplex (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format, including combinations thereof. Communication interface 760 may be configured to communicate with external devices, such as servers, user devices, or some other computing device.Atty Docket No. 0120-1137WO1

[0100] I / O device(s) 770 may include computer peripherals that facilitate the interaction between the user and computing system 700. Examples of I / O device(s) 770 may include keyboards, mice, trackpads, monitors, displays, printers, cameras, microphones, external storage devices, and the like.

[0101] Processing system 750 comprises microprocessor circuitry (e.g., at least one processor) and other circuitry that retrieves and executes operating software from storage system 745. Storage system 745 may include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for information storage, such as computer- readable instructions, data structures, program modules, or other data. Storage system 745 may be implemented as a single storage device, but may also be implemented across multiple storage devices or sub-systems. Storage system 745 may comprise additional elements, such as a controller to read operating software from the storage systems. Examples of storage media (also referred to as computer-readable storage media) include random access memory, read-only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof or any other type of storage media. In some implementations, the storage media may be non-transitory. In some instances, at least a portion of the storage media may be transitory. In no case is the storage media a propagated signal.

[0102] Processing system 750 is typically mounted on a circuit board that may hold the storage system. The operating software of storage system 745 comprises computer programs, firmware, or another form of machine-readable program instructions. The operating software of storage system 745 comprises ray application 724. The operating software on storage system 745 may include an operating system, utilities, drivers, network interfaces, applications, or other types of software. When read and executed by processing system 750, the operating software on storage system 745 directs computing system 700 to operate as described in the previously described FIGs 1-6.

[0103] In at least one implementation, ray application 724 directs processing system 750 to define a ray that is aligned along at least a portion of a user’s hand. This operation may be achieved by processing image data received from VO device(s) 770, such as one or more cameras, to perform skeletal tracking. From this tracking, a plurality of features or keypoints on the user’s hand and arm can be identified in three-dimensional space. A ray can then be defined based on the relative positions of two or more of these keypoints, such as a vector originatingAtty Docket No. 0120-1137WO1 from a wrist keypoint and pointing towards a metacarpophalangeal (MCP) joint keypoint, resulting in a ray that is substantially parallel to the user’s palm.

[0104] The processing system 750 can then be directed to display an indicator on a display, which can be one of the I / O device(s) 770, with the indicator’s position being based on the defined ray. In some examples, the ray’s behavior may be modified to enhance ergonomic control. For example, a CD gain may be applied to the user’s movements. The processing system 750 can be configured to track an angular displacement of a joint, such as the wrist relative to the forearm, and multiply this measured angle by a predetermined gain factor. This can amplify small physical motions into larger, more responsive movements of the indicator on the display.

[0105] In further implementations, a mixed joint gain controller model can be used. Ray application 724 may direct processing system 750 to identify a first movement, such as wrist flexion, and a second movement, such as forearm rotation. A first gain may be associated with the first movement, while a second, different gain may be associated with the second movement. A higher gain may be applied to forearm rotation for rapid, coarse navigation, and a lower gain may be applied to wrist movement for precise, fine-tuned adjustments. The scaled outputs can be combined to determine the indicator’s final position. Additionally, the ray may be subject to a parallel translation, shifting it to a new position while preserving its orientation. The indicator’s location can then be based on this translated ray, allowing a user to interact with content at eyelevel while keeping their arm in a comfortable, lowered position.

[0106] To illustrate, a user wearing an XR device, a component of computing system 700, may wish to select an application icon displayed on a virtual interface. With their arm in a comfortable, resting position at their side, the user first performs a small rotation of their forearm to move the indicator toward the icon. Ray application 724 directs processing system 750 to track this movement and apply the associated high gain factor, which translates the small physical motion into a large, sweeping movement of the indicator across the display, bringing it into the general vicinity of the target icon. To make the final, precise adjustment, the user then performs a slight flexion of their wrist. Processing system 750 applies the lower gain associated with wrist movement, resulting in a slower, more controlled motion that allows the user to accurately position the indicator directly over the icon. Finally, the user can perform a tap gesture with their thumb and index finger, which is detected by I / O device(s) 770 and interpreted as a selection command, thereby activating the icon.Atty Docket No. 0120-1137WO1

[0107] Below are example clauses associated with the present disclosure. The described clauses should not be considered exhaustive.

[0108] Clause 1. A method comprising: defining a ray aligned along at least a portion of a hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

[0109] Clause 2. The method of clause 1, wherein the portion of the user comprises a forearm of the user, wherein the relationship between the ray and the portion of the user comprises an angle between the ray and the forearm of the user, and wherein the modification comprises a gain associated with the angle.

[0110] Clause 3. The method of clause 2, further comprising: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

[0111] Clause 4. The method of clause 3, wherein the second gain is greater than the gain.

[0112] Clause 5. The method of any one of the preceding clauses, further comprising: identifying a translation of the ray, wherein displaying the indicator is further based on the translation.

[0113] Clause 6. The method of clause 5, wherein the translation is parallel to the ray.

[0114] Clause 7. The method of any one of the preceding clauses, wherein defining a ray aligned along at least a portion of the hand of the user comprises: receiving at least one image of the hand; identifying two or more features of the hand based on the at least one image; and defining the ray based on the two or more features of the hand.

[0115] Clause 8. The method of clause 7, wherein the two or more features comprise two or more of a wrist, a joint on the hand, or a fingertip.

[0116] Clause 9. The method of any one of the preceding clauses, further comprising: identifying an input based on a touch of a thumb to the hand.

[0117] Clause 10. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: defining a ray aligned along at least a portion of aAtty Docket No. 0120-1137WO1 hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

[0118] Clause 11. The computer-readable storage medium of clause 10, wherein the portion of the user comprises a forearm of the user, wherein the relationship between the ray and the portion of the user comprises an angle between the ray and the forearm of the user, and wherein the modification comprises a gain associated with the angle.

[0119] Clause 12. The computer-readable storage medium of clause 11, wherein the method further comprises: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

[0120] Clause 13. The computer-readable storage medium of clause 12, wherein the second gain is greater than the gain.

[0121] Clause 14. The computer-readable storage medium of any one of clauses 10 to 13, wherein the method further comprises: identifying a translation of the ray, wherein displaying the indicator is further based on the translation.

[0122] Clause 15. The computer-readable storage medium of clause 14, wherein the translation is parallel to the ray.

[0123] Clause 16. The computer-readable storage medium of any one of clauses 10 to 15, wherein defining a ray aligned along at least a portion of the hand of the user comprises: receiving at least one image of the hand; identifying two or more features of the hand based on the at least one image; and defining the ray based on the two or more features of the hand.

[0124] Clause 17. The computer-readable storage medium of clause 16, wherein the two or more features comprise two or more of a wrist, a joint on the hand, or a fingertip.

[0125] Clause 18. The computer-readable storage medium of any one of clauses 10 to 17, wherein the method further comprises: identifying an input based on a touch of a thumb to the hand.

[0126] Clause 19. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the at least one processor to perform a method, the methodAtty Docket No. 0120-1137WO1 comprising: defining a ray aligned along at least a portion of a hand of a user; identifying an angle between a forearm of the user and the ray; identifying a gain associated with the angle; and displaying an indicator on a display of a wearable device based on the gain associated with the angle.

[0127] Clause 20. The computing system of clause 19, wherein the method further comprises: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

[0128] In accordance with aspects of the disclosure, implementations of various techniques and methods described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0129] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the implementations. They have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementationsAtty Docket No. 0120-1137WO1 described herein can include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0130] It will be understood that, in the foregoing description, when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it may be directly on, connected or coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, directly connected to or directly coupled to another element, there are no intervening elements present. Although the terms directly on, directly connected to, or directly coupled to may not be used throughout the detailed description, elements that are shown as being directly on, directly connected or directly coupled can be referred to as such. The claims of the application, if any, may be amended to recite exemplary relationships described in the specification or shown in the figures.

[0131] As used in this specification, a singular form may, unless definitively indicating a particular case in terms of the context, include a plural form. Spatially relative terms (e.g., over, above, upper, under, beneath, below, lower, and so forth) are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. In some implementations, the relative terms above and below can, respectively, include vertically above and vertically below. In some implementations, the term adjacent can include laterally adjacent to or horizontally adjacent to.

Claims

Atty Docket No. 0120-1137WO1WHAT IS CLAIMED IS:

1. A method comprising: defining a ray aligned along at least a portion of a hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

2. The method of claim 1, wherein the portion of the user comprises a forearm of the user, wherein the relationship between the ray and the portion of the user comprises an angle between the ray and the forearm of the user, and wherein the modification comprises a gain associated with the angle.

3. The method of claim 2, further comprising: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

4. The method of claim 3, wherein the second gain is greater than the gain.

5. The method of any one of the preceding claims, further comprising: identifying a translation of the ray, wherein displaying the indicator is further based on the translation.

6. The method of claim 5, wherein the translation is parallel to the ray.

7. The method of any one of the preceding claims, wherein defining the ray aligned along at least the portion of the hand of the user comprises: receiving at least one image of the hand; identifying two or more features of the hand based on the at least one image; andAtty Docket No. 0120-1137WO1 defining the ray based on the two or more features of the hand.

8. The method of claim 7, wherein the two or more features comprise two or more of a wrist, a joint on the hand, or a fingertip.

9. The method of any one of the preceding claims, further comprising: identifying an input based on a touch of a thumb to the hand.

10. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: defining a ray aligned along at least a portion of a hand of a user; determining a modification to the ray based on a relationship between the ray and a portion of the user; and displaying an indicator on a display of a wearable device based on the modification.

11. The computer-readable storage medium of claim 10, wherein the portion of the user comprises a forearm of the user, wherein the relationship between the ray and the portion of the user comprises an angle between the ray and the forearm of the user, and wherein the modification comprises a gain associated with the angle.

12. The computer-readable storage medium of claim 11, wherein the method further comprises: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

13. The computer-readable storage medium of claim 12, wherein the second gain is greater than the gain.Atty Docket No. 0120-1137WO114. The computer-readable storage medium of any one of claims 10 to 13, wherein the method further comprises: identifying a translation of the ray, wherein displaying the indicator is further based on the translation.

15. The computer-readable storage medium of claim 14, wherein the translation is parallel to the ray.

16. The computer-readable storage medium of any one of claims 10 to 15, wherein defining the ray aligned along at least the portion of the hand of the user comprises: receiving at least one image of the hand; identifying two or more features of the hand based on the at least one image; and defining the ray based on the two or more features of the hand.

17. The computer-readable storage medium of claim 16, wherein the two or more features comprise two or more of a wrist, a joint on the hand, or a fingertip.

18. The computer-readable storage medium of any one of claims 10 to 17, wherein the method further comprises: identifying an input based on a touch of a thumb to the hand.

19. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the at least one processor to perform a method, the method comprising: defining a ray aligned along at least a portion of a hand of a user; identifying an angle between a forearm of the user and the ray; identifying a gain associated with the angle; andAtty Docket No. 0120-1137WO1 displaying an indicator on a display of a wearable device based on the gain associated with the angle.

20. The computing system of claim 19, wherein the method further comprises: identifying a second angle between the forearm of the user and another vector associated with another portion of the user; and identifying a second gain based on the second angle, wherein displaying the indicator is further based on the second gain.

Images