Human-computer interaction method and apparatus, extended reality device, and storage medium

Abstract

The present application provides a human-computer interaction method and apparatus, an extended reality device, and a storage medium. In the method of the present application, fingertip coordinates of a hand of a user are identified by means of a fingertip tracking algorithm; a target ray is fitted by using a ray direction from the eyes to the fingertip and the fingertip coordinates; a collision intersection point between the target ray and a screen interface is determined by means of collision detection; and a position of the user's intended operation is determined; and collision intersection point coordinates are converted into screen coordinates by means of a coordinate mapping algorithm. Thus, a correspondence between a user gesture and a control is achieved.

Description

Human-computer interaction methods, devices, extended reality equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 2024119753158, filed with the Chinese Patent Office on December 30, 2024, entitled "Human-Computer Interaction Method, Apparatus, Extended Reality Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method, apparatus, extended reality device and storage medium. Background Technology

[0003] In human-computer interaction technologies for extended reality devices (such as AR, VR, and MR devices), commonly used interaction methods include controllers, touchscreens, or gesture recognition. However, gesture recognition-based interaction technologies generally suffer from low recognition accuracy, especially in fingertip positioning and interaction with the virtual environment, where delays and errors occur.

[0004] Therefore, improving the interaction efficiency of extended reality devices has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a human-computer interaction method, apparatus, device, and storage medium, aiming to improve the interaction efficiency of extended reality devices.

[0006] In a first aspect, this application provides a human-computer interaction method, the method comprising:

[0007] Identify the coordinates of the user's fingertips;

[0008] Based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates, a target ray is fitted;

[0009] Detect the collision point between the target ray and the screen interface, and obtain the coordinates of the collision point;

[0010] Map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located to obtain the screen coordinates corresponding to the collision intersection point;

[0011] The screen coordinates are used to identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operations on the target control.

[0012] Secondly, this application also provides a human-computer interaction device, the human-computer interaction device comprising:

[0013] The fingertip coordinate recognition module is used to identify the coordinates of the user's fingertips.

[0014] A ray fitting module is used to fit a target ray based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates;

[0015] The collision intersection detection module is used to detect the collision intersection between the target ray and the screen interface and obtain the coordinates of the collision intersection.

[0016] The coordinate mapping module is used to map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, so as to obtain the screen coordinates corresponding to the collision intersection point;

[0017] The target control recognition module is used to identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operations on the target control.

[0018] Thirdly, this application also provides an extended reality device, the extended reality device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the human-computer interaction method described above.

[0019] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the human-computer interaction method described above.

[0020] This application provides a human-computer interaction method, apparatus, extended reality device, and storage medium. The method includes identifying the fingertip coordinates of a user's hand; fitting a target ray based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates; detecting the collision intersection point between the target ray and the screen interface, and obtaining the collision intersection point coordinates; mapping the collision intersection point coordinates to the plane coordinate system where the screen interface is located, and obtaining the screen coordinates corresponding to the collision intersection point; and identifying the target control pointed to by the screen coordinates in the screen interface, so as to facilitate interactive operation of the target control. Through the above methods, this application accurately identifies the fingertip coordinates of the user's hand using a fingertip tracking algorithm, providing accurate input points for subsequent interactions and ensuring the accuracy of the interaction; it fits the target ray using the ray direction from the user's eye to the fingertip and the fingertip coordinates to determine the user's interaction object, making the interaction more intuitive and natural; it determines the collision intersection point between the target ray and the screen interface through a collision detection algorithm, and converts the collision intersection point coordinates into screen coordinates through a coordinate mapping algorithm to determine the specific location of the user's intended operation, and then identifies the target control of the user's intended interaction based on the screen coordinates, realizing a precise correspondence between the user's gesture and the control, thereby improving the accuracy and efficiency of the interaction, making the interaction in the augmented reality environment more convenient and efficient. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a flowchart illustrating a first embodiment of a human-computer interaction method provided in this application.

[0023] Figure 2 is a schematic diagram of a collision detection structure provided in an embodiment of this application;

[0024] Figure 3 is a flowchart illustrating a second embodiment of a human-computer interaction method provided in this application.

[0025] Figure 4 is a structural schematic diagram of a first embodiment of a human-computer interaction device provided in this application;

[0026] Figure 5 is a schematic block diagram of an extended reality device provided in an embodiment of this application.

[0027] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0030] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0031] Please refer to Figure 1, which is a flowchart illustrating a first embodiment of a human-computer interaction method provided in this application.

[0032] As shown in Figure 1, the human-computer interaction method includes steps S101 to S104.

[0033] S101. Identify the coordinates of the user's fingertips;

[0034] In one embodiment, when a user wears an augmented reality device (such as AR glasses, VR glasses, etc.), an image or video stream of the user's hand is captured by a camera configured in the augmented reality device, and the fingertip coordinates of the user's hand are identified by a fingertip tracking algorithm.

[0035] Extended Reality (XR) devices are wearable devices that combine reality and virtuality through computer technology to create a virtual environment that allows for human-computer interaction. These include Augmented Reality (AR) devices, Virtual Reality (VR) devices, and Mixed Reality (MR) devices.

[0036] In one embodiment, the fingertip coordinates can be the coordinates of the fingertip of a specific finger, such as the fingertip of the index finger. In this case, the fingertip tracking algorithm identifies the fingertip coordinates of the user's index finger. Alternatively, the fingertip coordinates can be the coordinates of the fingertip of all the fingers detected on the user's hand. For example, if only three fingers of the user's hand are detected in the video stream captured by the camera, the fingertip tracking algorithm identifies the fingertip coordinates of those three fingers.

[0037] In one embodiment, the fingertip coordinates are three-dimensional coordinates (x, y, z) in the imaging space of the extended reality device.

[0038] S102. Fit the target ray based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates;

[0039] In one embodiment, as shown in Figure 2, within the imaging space of the augmented reality device, the identified fingertip coordinates are used as the starting point coordinates of the target ray, and the ray direction is the direction from which the augmented reality device points to the fingertip, thus fitting the target ray. For example, within the imaging space of the augmented reality device, the direction from the location coordinates of the augmented reality device (such as the device fitting center coordinates) to the fingertip can be used as the ray direction.

[0040] In one embodiment, the ray equation of the target ray is calculated based on the fingertip coordinates and the ray direction.

[0041] In one embodiment, the calculation of the ray equation is typically a vector from one point to another. In the imaging space of the extended reality device, the ray direction of the target ray can be represented as a direction vector from the user's viewpoint coordinates (which could be the midpoint of the line connecting the user's two eyes) to the user's fingertip coordinates. The ray extending along this direction vector, starting from the fingertip coordinates, is the target ray.

[0042] For example, suppose the user's viewpoint coordinates are E = (E x E y E z The user's fingertip coordinates are F = (Fx F y F z ). Direction vector of the target ray This can be calculated by subtracting the viewpoint coordinates from the fingertip coordinates:

[0043] To obtain the unit direction vector, the direction vector needs to be... Normalize it so that its length is 1.

[0044] First calculate Length:

[0045] Then, the unit direction vector for:

[0046] In the ray equation, the unit direction vector D unit Multiplying by the parameter t to obtain the position of any point on the ray, the ray equation can be expressed as: P(t) = P + t * D unit

[0047] Where P is the starting point of the target ray, i.e., the fingertip coordinate F, and P(t) represents the position coordinates of any parameter t on the target ray, where parameter t represents the position of the target ray along the unit direction vector D from the starting point P. unit The distance. Where, because D unit Let P be a unit vector and P be the starting point of the target ray. Therefore, the coordinate position of any point on the target ray can be expressed by setting the value of the parameter t. In other words, any point on the target ray uniquely corresponds to a parameter t.

[0048] S103. Detect the collision intersection point between the target ray and the screen interface, and obtain the coordinates of the collision intersection point;

[0049] Generally, collision detection algorithms can be used to determine whether two or more objects intersect or contact each other. In this embodiment, a collision detection algorithm is used to calculate whether there is a collision intersection point between the target ray and the plane containing the screen interface, and to calculate the coordinates of the collision intersection point.

[0050] For example, in this embodiment of the application, the ray equation can be substituted into the plane equation of the plane where the screen interface is located, and then the ray equation P(t)=P+t*D can be solved. unit The parameter t in the equation is the parameter value of the target ray from the fingertip coordinates to the plane where the screen interface is located. Substituting the parameter t into the ray equation and solving for the unique point coordinates is the collision intersection point coordinate.

[0051] It is understood that the collision detection algorithm can also use other calculation methods that can be applied to the embodiments of this application. For example, the ray collision detection function of OpenGL can be used to calculate the collision point coordinates, or the collision detection process can be optimized by the ray tracing algorithm to further reduce latency and improve the interactive experience.

[0052] Specifically, given the ray equation and the plane equation, the ray equation P(t) = P + t*D is obtained by... unit Substitute the values ​​into the plane equation to solve for the parameter t in the ray equation. If the value of the parameter t obtained is within a reasonable range (usually a non-negative number), it indicates that the target ray intersects the plane. Specifically, if substituting the ray equation into the plane equation yields a non-negative parameter t, it can be understood that extending along the ray direction from the fingertip coordinates, a point on the target ray will intersect the plane containing the screen interface. This intersection point is the coordinate of a point on the target ray obtained by substituting the parameter t into the ray equation. Because this point is the intersection of the target ray and the plane containing the screen interface, it is also located on the plane containing the screen interface, i.e., the collision intersection point coordinates.

[0053] This embodiment combines a fingertip tracking algorithm to identify the coordinates of the user's fingertips, fits a target ray based on the fingertip coordinates, and then performs collision detection with the screen interface to obtain the coordinates of the collision intersection point. This improves the accuracy and response speed of the interaction, allowing users to interact directly with the virtual interface through gestures, enhancing immersion and user experience, while reducing interaction latency and improving interaction efficiency, providing users with a more natural and intuitive interaction method.

[0054] Further, the plane equation of the screen interface is obtained; based on the ray equation corresponding to the target ray and the plane equation, the coordinates of the collision intersection point between the target ray and the screen interface are calculated.

[0055] Generally, the screen interface can be considered as a plane, which can usually be represented as: n·(p-p0)=0

[0056] Where n is the normal vector of the plane, p is a point on the plane, and p0 is any point on the plane.

[0057] In one embodiment, the ray equation is substituted into the plane equation to solve for the parameter t. That is, assuming the target ray intersects the plane containing the screen interface at a point P(t0), the coordinates of this intersection point on the target ray are expressed as: P(t0) = P + t0 * D unit

[0058] Simultaneously, this intersection point also lies on the plane where the screen interface is located. Substituting the coordinates of this intersection point into the plane equation, the parameter t0 is solved using the plane equation. Specifically, the solution process is as follows: n·((P+t0·D) unit )-p0)=0 n·(P-p0+t0·D unit )=0 n·(P-p0)+t0·(n·D unit )=0 t0·(n·D unit )=-n·(P-p0)

[0059] Where n is the normal vector of the plane, P is the apex coordinate, p0 is the coordinate of any point on the plane, and D... unit It is the unit direction vector of the target ray; if the calculated result of parameter t0 is non-negative, then the target ray intersects the plane containing the screen.

[0060] Substituting the value of t0 into the ray equation, calculate the coordinates of the collision intersection point: Intersection Point = P + t0 * D unit

[0061] Wherein, Intersection Point represents the coordinates of the collision intersection point.

[0062] In one embodiment, since the collision detection algorithm detects the collision intersection point between the target ray and the plane where the screen interface is located, but the screen interface has boundaries in the imaging space of the extended reality device, after calculating the collision intersection point coordinates, it is also necessary to detect whether the X-axis coordinates and Y-axis coordinates of the collision intersection point are within the effective range of the screen interface.

[0063] If the X-axis and Y-axis coordinates of the collision intersection point are within the valid range, it means that the user's fingertips and the screen interface have effective interaction, and the user can perform effective manipulation of the controls on the screen interface; if the X-axis and Y-axis coordinates of the collision intersection point are not within the valid range, it means that the user's fingertips and the screen interface have not achieved effective interaction, and the user cannot manipulate the controls on the screen interface.

[0064] Furthermore, based on the collision detection algorithm, the collision intersection point between the target ray and the screen interface is detected to obtain the coordinates of the undetermined intersection point; the effective coordinate area corresponding to the screen interface is obtained; when the coordinates of the undetermined intersection point are located within the effective coordinate area, the coordinates of the undetermined intersection point are determined as the coordinates of the collision intersection point.

[0065] In one embodiment, the effective coordinate region refers to the area on the screen interface that the user can effectively interact with, and the effective coordinate region defines the effective coordinate range of the screen interface on the X-axis and Y-axis in the planar coordinate system.

[0066] For example, the effective coordinate area can be determined based on the physical size of the screen or the user interface. For instance, if the screen interface is a projection interface of an external display device (such as a mobile phone, computer, etc.), the area coordinates of the effective coordinate area are calculated based on the projection ratio and the screen size of the external display device; if the screen interface is a visual display of the user interface within the field of view of an extended reality device, the area coordinates of the effective coordinate area are determined based on the display settings parameters of the extended reality device, such as the screen start coordinates, width, and height.

[0067] For example, the effective coordinate region can be a rectangular area (or other shapes) in the plane where the screen interface is located. When calculating the coordinates of the undetermined intersection point of the collision point between the target ray and the plane where the screen interface is located, it is determined whether the coordinates of the undetermined intersection point are located within the effective coordinate region.

[0068] If the coordinates of the undetermined intersection point are outside the valid coordinate area, it means that the user's fingertips cannot interact effectively with the screen interface at the current position; if the coordinates of the undetermined intersection point are within the valid coordinate area, it means that the user's fingertips can interact effectively with the screen interface at the current position, and the coordinates of the undetermined intersection point are determined as the collision intersection point coordinates.

[0069] Specifically, the coordinates of the undetermined intersection point are three-dimensional coordinates. The X-axis and Y-axis coordinates of the undetermined intersection point are compared with the boundary of the valid coordinate area. If both the X-axis and Y-axis coordinates of the undetermined intersection point are within the valid coordinate area of ​​the screen interface, then the undetermined intersection point coordinates are determined to be valid and identified as collision intersection point coordinates; otherwise, if either the X-axis or Y-axis coordinate of the undetermined intersection point is outside the valid coordinate area of ​​the screen interface, then the undetermined intersection point coordinates are invalid.

[0070] This embodiment calculates the coordinates of the collision intersection point between the target ray and the screen interface, and verifies whether the collision intersection point coordinates are within the effective coordinate area of ​​the screen interface. This ensures effective interaction between user gestures and the virtual interface, improves the accuracy and reliability of gesture control, and enables users to manipulate the controls on the screen more accurately, thereby improving the efficiency and practicality of the interaction.

[0071] S104. Map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located to obtain the screen coordinates corresponding to the collision intersection point;

[0072] In one embodiment, after collision detection is completed, the coordinates of the three-dimensional collision intersection point are mapped to the two-dimensional screen coordinates in the plane coordinate system where the screen interface is located through a coordinate mapping algorithm.

[0073] For example, the mapping process of the coordinate mapping algorithm may include: converting the collision intersection coordinates into clipping coordinates through view transformation and projection transformation, converting the clipping coordinates into normalized device coordinates (NDC) coordinates, and then converting the NDC coordinates into screen coordinates.

[0074] Specifically, view transformation refers to converting the collision intersection coordinates from the world coordinate system to the view coordinate system, that is, converting the collision intersection coordinates in the world coordinate system to point coordinates in the camera coordinate system. Projection transformation, through perspective projection or orthographic projection, converts the three-dimensional view coordinates into two-dimensional clipping coordinates. Then, the clipping coordinates are converted into NDC coordinates, which range from [-1,1]. Multiplying the NDC coordinates by the screen size (or viewport size) and adding the viewport offset, maps the range from [-1,1] to the actual pixel coordinates on the screen.

[0075] Generally, in the process of converting clipping coordinates to NDC coordinates, for the X-axis and Y-axis coordinates, the clipping coordinates are usually divided by the W coordinate (in homogeneous coordinates), and the result is scaled to the range of [-1, 1].

[0076] It is understood that the coordinate mapping process of the above coordinate mapping algorithm is a feasible solution already existing in the field. In short, this algorithm maps a point in one coordinate system to another through a series of mathematical transformations to meet specific application requirements. Although the specific implementation details may differ in different application scenarios and performance requirements across different application domains, the core principles and methodology are similar. For specific implementation processes, please refer to relevant existing technologies in the field; the specific calculation process is not described in detail in the embodiments of this application.

[0077] S105. Identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operation on the target control.

[0078] In one embodiment, after calculating the screen coordinates corresponding to the collision intersection point, corresponding UI interaction events can be triggered based on the controls in the control system (such as Android) interface according to the screen coordinates. These events could include Hover, Move, and Click. The control system then responds to the current user's actions based on the triggered UI interaction events, such as selecting a button or swiping a list.

[0079] Furthermore, based on the fingertip tracking algorithm, the fingertip coordinate movement of the user's hand is calculated; based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, the fingertip coordinate movement is mapped to the plane coordinate system where the screen interface is located to obtain the screen coordinate movement; based on the screen coordinate movement, the corresponding cursor movement operation is performed on the screen interface.

[0080] During coordinate mapping, the mapped screen coordinates can be displayed as a cursor in the screen interface to intuitively represent the position of the user's fingertip coordinates in the coordinate system of the screen interface, making it easier for the user to perform precise interactive operations on the controls in the screen interface.

[0081] In one embodiment, a fingertip tracking algorithm is used to calculate the change in coordinate distance between two consecutively acquired fingertip coordinates, thus obtaining the fingertip coordinate movement during the two consecutive acquisition processes. For example, if the first acquired fingertip coordinates are (x1, y1, z1) and the second acquired fingertip coordinates are (x2, y2, z2), then the fingertip coordinate movement is (x2-x1, y2-y1, z2-z1). Based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, the fingertip coordinate movement is mapped to the plane coordinate system where the screen interface is located, and converted into the screen coordinate movement in the screen interface.

[0082] In one embodiment, the fingertip coordinate movement can be calculated first, and then mapped to screen coordinate movement. Alternatively, the fingertip coordinates can be mapped to screen coordinates first, and then the screen coordinate movement between the two mappings can be calculated.

[0083] In one embodiment, since screen coordinates are two-dimensional, the screen coordinate movement can consist of two parts: X-axis movement and Y-axis movement. Therefore, it is only necessary to map the X-axis and Y-axis movement of the fingertip coordinates to the plane coordinate system where the screen interface is located to obtain the screen coordinate movement. Alternatively, the X-axis and Y-axis coordinates of the two collected fingertip coordinates can be mapped to the plane coordinate system to obtain the screen coordinates corresponding to the two collected fingertip coordinates. Then, the movement of the two screen coordinates along the X-axis and the movement along the Y-axis can be calculated to obtain the screen coordinate movement.

[0084] The mapping of coordinate movement can be performed based on the size conversion ratio between the pixel size of the 3D coordinate system in the extended reality device and the pixel size of the planar coordinate system where the screen interface is located. For example, if the size conversion ratio between the pixel size of the 3D coordinate system in the extended reality device and the pixel size of the planar coordinate system where the screen interface is located is 2:1, that is, a point moving two pixels along the X-axis in the 3D coordinate system is mapped to a point moving one pixel along the X-axis in the planar coordinate system where the screen interface is located. Accordingly, when mapping the X / Y axis coordinate movement, the X-axis and Y-axis coordinate movement in the fingertip coordinates also need to be multiplied by the size conversion ratio to adapt to the coordinate size in the planar coordinate system where the screen interface is located, thus mapping them to the X-axis and Y-axis coordinate movement in the planar coordinate system.

[0085] This embodiment uses a coordinate mapping algorithm to convert the coordinates of the three-dimensional collision intersection point into two-dimensional coordinates in the screen coordinate system, realizing precise interaction between user gestures and screen interface controls. It simplifies the connection between gesture recognition and screen operation, enabling the control system to respond to user gesture operations more quickly and accurately. This not only improves the accuracy of gesture control but also enhances the intuitiveness and convenience of the user interaction experience.

[0086] This embodiment provides a human-computer interaction method. The method uses a fingertip tracking algorithm to accurately identify the user's fingertip coordinates, providing accurate input points for subsequent interactions and ensuring accuracy. It fits a target ray using the ray direction from the user's eye to the fingertip and the fingertip coordinates to determine the user's interaction object, making the interaction more intuitive and natural. A collision detection algorithm determines the collision intersection point between the target ray and the screen interface, pinpointing the specific location of the user's intended operation, which helps achieve faster response and more precise control, enhancing the real-time nature of the interaction and improving efficiency. Finally, a coordinate mapping algorithm converts the collision intersection point coordinates into screen coordinates, achieving a precise correspondence between the user's gestures and controls, thereby improving the accuracy and efficiency of the interaction and making interaction in augmented reality environments more convenient and efficient.

[0087] Please refer to Figure 3, which is a flowchart illustrating a second embodiment of a human-computer interaction method provided in this application.

[0088] As shown in Figure 3, based on the embodiment shown in Figure 1 above, after step S104, the method further includes:

[0089] S201. Obtain the coordinate region corresponding to at least one control in the screen interface;

[0090] In one embodiment, the screen interface can be used as a coordinate plane based on its resolution. For example, the top-left corner of the screen interface can be used as the origin (0,0), with the horizontal direction as the X-axis and the vertical direction as the Y-axis. The width of the screen interface defines the maximum value of the X-axis, and the height of the screen interface defines the maximum value of the Y-axis. For instance, if the screen resolution is 1920*1080, then the range of the X-axis is 0 to 1920, and the range of the Y-axis is 0 to 1080.

[0091] The parameters such as the origin of the coordinate system of the screen interface can be set according to actual needs or user habits. For example, the center of the screen can be used as the origin to construct the coordinate system, or other points in the screen interface can be used as the origin.

[0092] In one embodiment, in the coordinate system of the screen interface, the coordinate region of each control is recorded based on the position of the area where each control is located relative to the origin. For example, for the icon control of an application app, the coordinates of the four corners of the icon control can be collected, and the area formed by the coordinates of the four corners can be used as the coordinate region corresponding to the icon control of the application app. For an input method keyboard, the coordinates of the intersection points of the edges of the input method keyboard can be collected to determine the coordinate region of the input method keyboard. For the keys in the input method keyboard, the coordinate region corresponding to each key control can be calculated based on the area size of the input method keyboard, the coordinate region, the size and distribution of the keys.

[0093] S202. Determine the target control based on the coordinate region corresponding to the screen coordinates;

[0094] In one embodiment, screen coordinates are matched with the coordinate regions of each control to determine whether the screen coordinates are located within the coordinate region corresponding to a certain control. If the screen coordinates are located within the coordinate region corresponding to a certain control, the control is identified as the target control, facilitating further interactive operations by the user.

[0095] In one embodiment, if the screen coordinates are located in a blank area, that is, there are no controls at the position corresponding to the screen coordinates, then interactive operations can be performed on the entire screen interface. For example, a swipe operation can be performed on the screen interface to switch the currently displayed page to the next displayed page hidden in the swipe direction.

[0096] This embodiment achieves precise control positioning and convenient user interaction by converting the screen interface into a coordinate plane and recording the coordinate area of ​​the controls, thereby improving the accuracy and efficiency of operation and enhancing the user experience.

[0097] S203. Obtain the control operation gesture library corresponding to the target control, wherein the control operation gesture library includes at least one control operation gesture;

[0098] In one embodiment, control operation gestures refer to gestures used by the user when interacting with a control, such as clicking, swiping, and long-pressing. These control operation gestures can be recognized and converted into corresponding control operation instructions.

[0099] In one embodiment, the control gesture library may include predefined control gestures; it may also allow users to customize control gestures and save them to the control gesture library, or modify and replace predefined control gestures.

[0100] Furthermore, at least one control operation instruction corresponding to the control in the screen interface is obtained; the control operation gesture corresponding to each control operation instruction is preset to construct a control operation gesture library corresponding to the control.

[0101] In one embodiment, the control gesture library includes at least one predefined set of control gestures, each control gesture corresponding to a control operation instruction. Different control operation instructions can correspond to different control gestures for differentiation; different control operation instructions can also correspond to the same control gesture. For example, if a certain control corresponds to a unique details page, the user can tap once to display the unique details page and tap again to hide the unique details page.

[0102] For example, the interactive operations that each control can perform can be obtained in advance, such as selection and editing operations. Then, the control operation gestures corresponding to each interactive operation are defined, and the control operation instructions corresponding to each control operation gesture are determined, creating a control operation gesture library for each control. For example, a single click gesture corresponds to a selection operation instruction, and a long press gesture corresponds to an editing operation instruction, etc.

[0103] Specifically, for an input method keyboard, tapping a key on the keyboard can be defined as a gesture for inputting the corresponding character. The extended reality device can then recognize this tap gesture using a gesture recognition algorithm and convert it into the corresponding character input command.

[0104] This embodiment utilizes a control gesture library to make user interaction with controls more intuitive and personalized. The library not only includes predefined gestures but also supports user-defined gestures, improving operational flexibility and convenience. Furthermore, by matching gestures with control operation commands, user intent can be more accurately recognized, thereby enhancing user experience and interaction efficiency.

[0105] S204. Recognize the control operation gesture corresponding to the user's gesture action, generate control operation instructions, and perform corresponding control operations on the target control according to the control operation instructions.

[0106] In one embodiment, gesture recognition algorithms such as machine learning and deep learning can be used to recognize gesture actions. For example, MediaPipe's gesture recognition API can be used to analyze the coordinates of key points on the hand and recognize gesture actions. Then, by analyzing the gesture actions, the corresponding control operation gestures of the user's gesture actions can be identified.

[0107] For example, a single tap can be recognized as a single touch without movement for a short period of time; a double tap is two quick taps; a long press is touching and holding for a period of time; a swipe is touching and moving the finger; a drag is long pressing and moving the finger; and a pinch is touching and moving two fingers at the same time.

[0108] Furthermore, based on a gesture recognition algorithm, a first gesture feature of the gesture action is identified; based on a feature matching algorithm, the first gesture feature is matched with the second gesture feature of each control operation gesture to generate the control operation instruction corresponding to the control operation gesture with the highest matching degree.

[0109] For example, the MediaPipe gesture detection model can be used to recognize gestures, providing 3D coordinates of key points of the fingers (such as fingertips, knuckles, etc.), and recognizing the first gesture feature of the gesture action by acquiring hand key point (landmark) information.

[0110] In one embodiment, the control operation gesture library can construct control operation gestures corresponding to each control operation instruction, and define second gesture features such as key point patterns or coordinate sequences corresponding to each control operation gesture.

[0111] In one embodiment, a feature matching algorithm is used to calculate the feature similarity between the first gesture feature and the second gesture feature of each of the control operation gestures, and the second gesture feature with the highest matching degree is selected. Then, the user's gesture action is identified as the control operation gesture corresponding to the second gesture feature, and a control operation instruction corresponding to the control operation gesture is generated. The corresponding operation is performed on the target control according to the generated control operation instruction.

[0112] For example, a feature matching algorithm such as the Brute-Force matching algorithm is used to match the first gesture feature (i.e., the feature descriptor of the user's gesture) with the second gesture feature (i.e., the feature descriptor of the preset gesture) in the control operation gesture library. The Brute-Force matching algorithm finds the best matching feature point pair by calculating the similarity of each pair of feature points. For each matched feature point pair, the distance between the feature descriptors is calculated to express the feature similarity. Euclidean distance and Hamming distance can be used for feature similarity calculation. K-Nearest Neighbor (KNN) matching and comparison threshold filtering can be used to filter the matching results. For example, for each feature point, the two closest matching points are found and the ratio of their distances is calculated. Based on the filtered matching results, the similarity scores of all matching pairs are compared to select the feature point pair with the highest matching degree, thereby determining the control operation gesture corresponding to the gesture action.

[0113] In one embodiment, after the gesture action is recognized, a corresponding control operation instruction is generated, and the target control is operated according to the generated control operation instruction, such as clicking or swiping.

[0114] For example, suppose a user switches between app interfaces using a swipe gesture. The camera can capture the swipe, a gesture recognition algorithm can identify it as a swipe, generate control commands to switch interfaces, and execute the interface switch. In this way, the user can control the switching of app interfaces using gestures.

[0115] This embodiment combines coordinate mapping and gesture recognition to map fingertip coordinates to screen coordinates, accurately identifying target controls. By using a gesture recognition algorithm, it identifies the user's gestures and converts them into control operation commands, thereby achieving efficient control of the target controls. This improves the naturalness and intuitiveness of user interaction, enhances the flexibility and response speed of operation, and provides users with a more convenient and intuitive interactive experience.

[0116] Please refer to Figure 4, which is a schematic diagram of the structure of a first embodiment of a human-computer interaction device provided in this application. The human-computer interaction device is used to perform the aforementioned human-computer interaction method.

[0117] As shown in Figure 4, the human-computer interaction device 300 includes: a fingertip coordinate recognition module 301, a ray fitting module 302, a collision intersection detection module 303, a coordinate mapping module 304, and a target control recognition module 305.

[0118] The fingertip coordinate recognition module 301 is used to recognize the fingertip coordinates of the user's hand;

[0119] The ray fitting module 302 is used to fit a target ray based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates;

[0120] The collision intersection detection module 303 is used to detect the collision intersection between the target ray and the screen interface and obtain the coordinates of the collision intersection.

[0121] The coordinate mapping module 304 is used to map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, so as to obtain the screen coordinates corresponding to the collision intersection point.

[0122] The target control recognition module 305 is used to recognize the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operation of the target control.

[0123] In one embodiment, the collision intersection detection module 303 includes:

[0124] A plane equation acquisition unit is used to acquire the plane equation of the screen interface;

[0125] The collision intersection coordinate calculation unit is used to calculate the collision intersection coordinates between the target ray and the screen interface based on the ray equation corresponding to the target ray and the plane equation.

[0126] In one embodiment, the collision intersection detection module 303 further includes:

[0127] The undetermined intersection point coordinate detection unit is used to detect the collision intersection point between the target ray and the screen interface based on the collision detection algorithm, and obtain the coordinates of the undetermined intersection point.

[0128] A valid coordinate region acquisition unit is used to acquire the valid coordinate region corresponding to the screen interface;

[0129] The collision intersection point coordinate determination unit is used to determine the coordinates of the undetermined intersection point as the collision intersection point coordinates when the coordinates of the undetermined intersection point are located within the effective coordinate area.

[0130] In one embodiment, the human-computer interaction device 300 further includes a coordinate movement mapping module, comprising:

[0131] The coordinate movement calculation unit is used to calculate the coordinate movement of the user's fingertips based on the fingertip tracking algorithm.

[0132] The screen coordinate movement acquisition unit is used to map the fingertip coordinate movement to the plane coordinate system where the screen interface is located based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, so as to obtain the screen coordinate movement.

[0133] The movement operation execution unit is used to perform a corresponding cursor movement operation on the screen interface based on the screen coordinate movement amount.

[0134] In one embodiment, the human-computer interaction device 300 further includes a control operation instruction generation module, comprising:

[0135] The control coordinate area acquisition unit is used to acquire the coordinate area corresponding to at least one control on the screen interface.

[0136] The target space determination unit is used to determine the target control based on the coordinate region corresponding to the screen coordinates;

[0137] The price control operation gesture library acquisition unit is used to acquire the control operation gesture library corresponding to the target control, wherein the control operation gesture library includes at least one control operation gesture;

[0138] The control operation instruction generation unit is used to identify the control operation gesture corresponding to the user's gesture action, generate control operation instructions, and perform corresponding control operations on the target control according to the control operation instructions.

[0139] In one embodiment, the control operation instruction generation module further includes:

[0140] A control operation instruction acquisition unit is used to acquire at least one control operation instruction corresponding to a control in the screen interface.

[0141] The control operation gesture library construction unit is used to preset the control operation gestures corresponding to each control operation instruction and construct the control operation gesture library corresponding to the control.

[0142] In one embodiment, the coordinate mapping module 304 includes:

[0143] The clipping coordinate transformation unit is used to convert the collision intersection coordinates into clipping coordinates through view transformation and projection transformation;

[0144] The NDC coordinate transformation unit is used to convert the cutting coordinates into standardized device NDC coordinates;

[0145] The screen coordinate transformation unit is used to convert NDC coordinates to screen coordinates.

[0146] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the device and each module described above can be referred to the corresponding processes in the aforementioned human-computer interaction method embodiments, and will not be repeated here.

[0147] The apparatus provided in the above embodiments can be implemented as a computer program that can run on the extended reality device shown in FIG5.

[0148] Please refer to Figure 5, which is a schematic block diagram of an extended reality device provided in an embodiment of this application. The extended reality device may be a server.

[0149] Referring to Figure 5, the extended reality device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0150] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any human-computer interaction method.

[0151] The processor provides computing and control capabilities to support the operation of the entire extended reality device.

[0152] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any human-computer interaction method.

[0153] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the extended reality device to which the present application is applied. A specific extended reality device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0154] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0155] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0156] Identify the coordinates of the user's fingertips;

[0157] Based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates, a target ray is fitted;

[0158] Detect the collision point between the target ray and the screen interface, and obtain the coordinates of the collision point;

[0159] Map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located to obtain the screen coordinates corresponding to the collision intersection point;

[0160] The screen coordinates are used to identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operations on the target control.

[0161] In one embodiment, when the processor implements the collision detection algorithm to detect the collision intersection point between the target ray and the screen interface and obtain the coordinates of the collision intersection point, it is configured to:

[0162] Obtain the plane equation of the screen interface;

[0163] Based on the ray equation corresponding to the target ray and the plane equation, calculate the coordinates of the collision intersection point between the target ray and the screen interface.

[0164] In one embodiment, when the processor implements the collision detection algorithm to detect the collision intersection point between the target ray and the screen interface and obtain the coordinates of the collision intersection point, it is also configured to:

[0165] Based on the collision detection algorithm, the collision intersection point between the target ray and the screen interface is detected, and the coordinates of the undetermined intersection point are obtained;

[0166] Obtain the valid coordinate area corresponding to the screen interface;

[0167] When the coordinates of the undetermined intersection point are located within the effective coordinate area, the coordinates of the undetermined intersection point are determined as the coordinates of the collision intersection point.

[0168] In one embodiment, after implementing the coordinate mapping algorithm to map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, and obtaining the screen coordinates corresponding to the collision intersection point, the processor is further configured to implement:

[0169] Based on the aforementioned fingertip tracking algorithm, the movement of the user's fingertip coordinates is calculated;

[0170] Based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, the fingertip coordinate movement is mapped to the plane coordinate system where the screen interface is located to obtain the screen coordinate movement.

[0171] Based on the screen coordinate movement, the corresponding cursor movement operation is performed on the screen interface.

[0172] In one embodiment, after implementing the coordinate mapping algorithm to map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, and obtaining the screen coordinates corresponding to the collision intersection point, the processor is further configured to implement:

[0173] Get the coordinate region corresponding to at least one control on the screen interface;

[0174] The target control is determined based on the coordinate region corresponding to the screen coordinates;

[0175] Obtain the control operation gesture library corresponding to the target control, wherein the control operation gesture library includes at least one control operation gesture;

[0176] The system identifies the control operation gesture corresponding to the user's gesture action, generates control operation instructions, and performs corresponding control operations on the target control according to the control operation instructions.

[0177] In one embodiment, before implementing the acquisition of the control gesture library corresponding to the target control, wherein the control gesture library includes at least one control gesture, the processor is further configured to implement:

[0178] Obtain at least one control operation instruction corresponding to a control in the screen interface;

[0179] Pre-set the control operation gestures corresponding to each control operation instruction, and construct a control operation gesture library for each control.

[0180] In one embodiment, when the processor implements the step of mapping the coordinates of the collision intersection point to the coordinate system of the screen interface to obtain the screen coordinates corresponding to the collision intersection point, it is configured to:

[0181] The collision intersection coordinates are converted into clipping coordinates through view transformation and projection transformation;

[0182] Convert the cutting coordinates to standardized device NDC coordinates;

[0183] Convert the NDC coordinates to the screen coordinates.

[0184] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the human-computer interaction methods provided in the embodiments of this application.

[0185] The computer-readable storage medium can be an internal storage unit of the extended reality device described in the foregoing embodiments, such as the hard drive or memory of the extended reality device. Alternatively, the computer-readable storage medium can be an external storage device of the extended reality device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the extended reality device.

[0186] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A human-computer interaction method, the method comprising: Identify the coordinates of the user's fingertips; Based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates, a target ray is fitted; Detect the collision point between the target ray and the screen interface, and obtain the coordinates of the collision point; Map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located to obtain the screen coordinates corresponding to the collision intersection point; The screen coordinates are used to identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operations on the target control.

2. The human-computer interaction method according to claim 1, wherein, The collision detection algorithm is used to detect the collision intersection point between the target ray and the screen interface, and obtain the coordinates of the collision intersection point, including: Obtain the plane equation of the screen interface; Based on the ray equation corresponding to the target ray and the plane equation, calculate the coordinates of the collision intersection point between the target ray and the screen interface.

3. The human-computer interaction method according to claim 2, wherein, The calculation of the collision intersection coordinates between the target ray and the screen interface based on the ray equation corresponding to the target ray and the plane equation includes: Substitute the ray equation into the plane equation to solve for the parameters in the ray equation; If the parameter is non-negative, substitute the parameter into the ray equation to calculate the coordinates of the collision intersection point.

4. The human-computer interaction method according to claim 1, wherein, The method of detecting the collision intersection point between the target ray and the screen interface based on the collision detection algorithm and obtaining the coordinates of the collision intersection point also includes: Based on the collision detection algorithm, the collision intersection point between the target ray and the screen interface is detected, and the coordinates of the undetermined intersection point are obtained; Obtain the valid coordinate area corresponding to the screen interface; When the coordinates of the undetermined intersection point are located within the effective coordinate area, the coordinates of the undetermined intersection point are determined as the coordinates of the collision intersection point.

5. The human-computer interaction method according to claim 4, wherein, The step of determining the coordinates of the undetermined intersection point as the collision intersection point coordinates when the coordinates of the undetermined intersection point are located within the effective coordinate area includes: If the X-axis and Y-axis coordinates of the undetermined intersection point are both within the valid coordinate area corresponding to the screen interface, then the undetermined intersection point coordinates are determined to be the collision intersection point coordinates.

6. The human-computer interaction method according to claim 1, wherein, After obtaining the screen coordinates corresponding to the collision intersection point by mapping the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located based on the coordinate mapping algorithm, the method further includes: Based on the aforementioned fingertip tracking algorithm, the movement of the user's fingertip coordinates is calculated; Based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, the fingertip coordinate movement is mapped to the plane coordinate system where the screen interface is located to obtain the screen coordinate movement. Based on the screen coordinate movement, the corresponding cursor movement operation is performed on the screen interface.

7. The human-computer interaction method according to claim 6, wherein, The calculation of the fingertip coordinate movement of the user's hand based on the fingertip tracking algorithm includes: Based on the aforementioned fingertip tracking algorithm, the fingertip coordinates are obtained from two consecutive acquisitions; Calculate the change in coordinate distance between the fingertip coordinates in two consecutive acquisitions to obtain the amount of fingertip coordinate movement during the two consecutive acquisitions.

8. The human-computer interaction method according to claim 7, wherein, The step of mapping the fingertip coordinate movement to the plane coordinate system where the screen interface is located, based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, to obtain the screen coordinate movement includes: Based on the coordinate mapping relationship between the collision intersection coordinates and the screen coordinates, the fingertip coordinates collected in two consecutive transactions are mapped to screen coordinates; Calculate the screen coordinate shift between the two mappings.

9. The human-computer interaction method according to claim 1, wherein, After obtaining the screen coordinates corresponding to the collision intersection point by mapping the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located based on the coordinate mapping algorithm, the method further includes: Get the coordinate region corresponding to at least one control on the screen interface; The target control is determined based on the coordinate region corresponding to the screen coordinates; Obtain the control operation gesture library corresponding to the target control, wherein the control operation gesture library includes at least one control operation gesture; The system identifies the control operation gesture corresponding to the user's gesture action, generates control operation instructions, and performs corresponding control operations on the target control according to the control operation instructions.

10. The human-computer interaction method according to claim 9, wherein, Before obtaining the control gesture library corresponding to the target control, wherein the control gesture library includes at least one control gesture, the method further includes: Obtain at least one control operation instruction corresponding to a control in the screen interface; Pre-set the control operation gestures corresponding to each control operation instruction, and construct a control operation gesture library for each control.

11. The human-computer interaction method according to claim 10, wherein, The preset control operation gestures corresponding to each of the control operation instructions are used to construct a control operation gesture library for each control, including: Obtain the interactive operations that can be performed for each of the aforementioned controls; Define the control operation gesture corresponding to each interactive operation; Determine the control operation instructions corresponding to each control operation gesture to create a control operation gesture library for each control.

12. The human-computer interaction method according to claim 9, wherein, Determining the target control based on the coordinate region corresponding to the screen coordinates includes: The screen coordinates are matched with the coordinate regions of each control to determine whether the screen coordinates are located within the coordinate region corresponding to any control. If the screen coordinates are located within the coordinate area corresponding to any control, then the control is identified as the target control.

13. The human-computer interaction method according to claim 12, wherein, After performing coordinate matching between the screen coordinates and the coordinate regions of each control to determine whether the screen coordinates are located within the coordinate region corresponding to any control, the method further includes: If there are no controls at the location corresponding to the screen coordinates, then interactive operations are performed on the screen interface.

14. The human-computer interaction method according to claim 9, wherein, The process of recognizing the user's gesture and generating control operation instructions includes: Based on a gesture recognition algorithm, the first gesture feature of the gesture action is identified; Based on the feature matching algorithm, the first gesture feature is matched with the second gesture feature of each of the control operation gestures to generate the control operation instruction corresponding to the control operation gesture with the highest matching degree.

15. The human-computer interaction method according to claim 14, wherein, Before generating the control operation instruction corresponding to the control operation gesture with the highest matching degree by performing feature matching between the first gesture feature and the second gesture feature of each of the control operation gestures based on the feature matching algorithm, the method further includes: In the control operation gesture library, the control operation gestures corresponding to each control operation instruction are constructed, and the second gesture features corresponding to each control operation gesture are defined.

16. The human-computer interaction method according to claim 14, wherein, The second gesture feature includes key point patterns or coordinate sequences.

17. The human-computer interaction method according to claim 1, wherein, The step of mapping the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, to obtain the screen coordinates corresponding to the collision intersection point, includes: The collision intersection coordinates are converted into clipping coordinates through view transformation and projection transformation; Convert the cutting coordinates to standardized device NDC coordinates; Convert the NDC coordinates to the screen coordinates.

18. A human-computer interaction device, the human-computer interaction device comprising: The fingertip coordinate recognition module is used to identify the coordinates of the user's fingertips. A ray fitting module is used to fit a target ray based on the ray direction from the extended reality device to the user's hand and the fingertip coordinates; The collision intersection detection module is used to detect the collision intersection between the target ray and the screen interface and obtain the coordinates of the collision intersection. The coordinate mapping module is used to map the coordinates of the collision intersection point to the plane coordinate system where the screen interface is located, so as to obtain the screen coordinates corresponding to the collision intersection point; The target control recognition module is used to identify the target control that the screen coordinates point to in the screen interface, so as to facilitate interactive operations on the target control.

19. An extended reality device, the extended reality device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the steps of the human-computer interaction method as described in any one of claims 1 to 17.

20. A computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the human-computer interaction method as described in any one of claims 1 to 17.

Images