Device control method, remote control device and display device

Abstract

A device control method, a remote control device, and a display device, relating to the technical field of terminals. The accuracy of controlling display devices by means of remote control devices can be improved. The method comprises: when the height of a user's finger from a remote control device is a first height, acquiring first position information corresponding to the projection position of the user's finger on the remote control device; and then, on the basis of the first position information and preset movement information corresponding to the first position information, determining a second position on a display device, wherein the second position is used for determining an object to be controlled on the display device, a communication connection is established between the remote control device and the display device, and the remote control device is used for controlling the display device.

Description

Equipment operation methods, remote control devices and display devices

[0001] This application claims priority to Chinese patent application filed on March 13, 2025, with application number 202510301194.7 and entitled "Equipment Control Method, Remote Control Device and Display Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal technology, and in particular to device control methods, remote control devices, and display devices. Background Technology

[0003] Currently, users can perform operations on the remote control device to remotely control the display device. The remote control device supports both contact-based and hovering operations. Contact-based operations refer to actions performed in contact with the remote control device. Hovering operations refer to actions performed without contact with the remote control device. When a user's finger hovers above the remote control device, a virtual finger is mapped onto the display device, and this virtual finger follows the user's finger on the display device. When a user's finger falls from above the remote control device to perform a click operation to control the display device, the virtual finger on the display device moves according to the fall of the user's finger. This may lead to errors in the control of the display device and lower control accuracy. Summary of the Invention

[0004] This application provides a device control method, a remote control device, and a display device, which can improve the accuracy of controlling the display device through the remote control device.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a device control method is provided, the method comprising: when a user's finger is at a height of a first distance from a remote control device, acquiring first position information corresponding to the projection position of the user's finger on the remote control device; determining a second position on a display device based on the first position information and preset movement information corresponding to the first position information, the second position being used to determine an object to be controlled on the display device; wherein the remote control device and the display device have a communication connection, and the remote control device is used to control the display device.

[0007] Based on the above technical solution, when the user's finger descends to a certain height from the remote control device, the projected position of the user's finger on the remote control device at that current height is not directly mapped to the position on the display device; that is, the position on the display device is no longer controlled to follow the user's finger. Instead, the position on the display device is predicted based on the position directly mapped from the user's finger's projected position on the remote control device, and the corresponding preset movement information. This predicts the position on the display device, i.e., the object to be manipulated on the display device. This solves the problem of low accuracy when a user's finger falls from above the remote control device to perform a click operation to control the display device. During the descent, the user's finger bends, causing the projected position on the remote control device to shift, resulting in the virtual finger on the display device also moving, leading to lower control accuracy. This improves the accuracy of controlling the display device via the remote control device.

[0008] In one possible design, when the height of the user's finger from the remote control device is a first height, acquiring first position information corresponding to the projection position of the user's finger on the remote control device includes: when the first height is less than or equal to a first height threshold, acquiring first position information corresponding to the projection position of the user's finger on the remote control device when the height of the user's finger from the remote control device is the first height threshold.

[0009] Thus, when the height of the user's finger from the remote control device is less than or equal to a first height threshold, it indicates a high probability that the user intends to perform a click operation on the remote control device to manipulate an object on the display device. Furthermore, when the height of the user's finger from the remote control device is less than or equal to the first height threshold, i.e., based on the first position information corresponding to the projection position of the user's finger on the remote control device at the first height threshold, and the preset movement information corresponding to this first position information, the position of the object to be manipulated can be predicted, further improving the accuracy of controlling the display device via the remote control device.

[0010] In one possible design, the method further includes: when the height of the user's finger from the remote control device is greater than or equal to the first height threshold, determining a third position corresponding to the projection position of the user's finger on the remote control device as the position of the object to be manipulated on the display device.

[0011] Thus, when the user's finger is at a height greater than or equal to the first height threshold from the remote control device, it indicates a low probability that the user intends to manipulate an object on the display device. Therefore, when the user's finger is at a height greater than or equal to the first height threshold, directly mapping the position of the user's finger on the remote control device to the position on the display device—that is, controlling the mapped position on the display device to follow the movement of the user's finger—can improve the accuracy of controlling the display device via the remote control.

[0012] It is understood that, in the embodiments of this application, the position on the display device directly mapped from the projection position of the user's finger on the remote control device can refer to the position on the display device determined based on the projection position of the user's finger on the remote control device and the size ratio between the remote control device and the display device.

[0013] In one possible design, the preset movement information includes at least one of a preset movement distance and a preset movement direction.

[0014] In one possible design, the method further includes: when the user's finger is at a second height from the remote control device, acquiring second position information corresponding to the projection position of the user's finger on the remote control device; the first position represented by the first position information and the second position represented by the second position information are located in different display areas on the display device, or the user operation corresponding to the first position information is different from the user operation corresponding to the second position information, or the finger used by the user when acquiring the first position information is different from the finger used by the user when acquiring the second position information; and the preset movement information corresponding to the first position information is different from the preset movement information corresponding to the second position information.

[0015] Thus, when the user's finger is projected at different positions on the remote control device, the position on the display device mapped based on that projection position may be in different areas of the display device, and the preset movement information corresponding to these positions on the display devices in different areas is different.

[0016] Alternatively, the user operation corresponding to the first location information can refer to an operation used to manipulate the object to be manipulated, determined based on the first location information and the preset movement information corresponding to the first location information. The user operation corresponding to the second location information can refer to an operation used to manipulate the object to be manipulated, determined based on the second location information and the preset movement information corresponding to the second location information. When the operations used to manipulate the object to be manipulated are different, the preset movement information corresponding to the location information used to determine the object to be manipulated can be different.

[0017] Alternatively, when the user uses different fingers to obtain position information (such as first position information and second position information) to determine the position on the display device, the corresponding preset movement information can also be different.

[0018] In this way, the preset movement information obtained by taking into account the above factors is more accurate, which can further improve the accuracy of controlling the display device through the remote control device.

[0019] In one possible design, after determining a second position on the display device based on the first position information and preset movement information corresponding to the first position information, the method further includes: manipulating the object to be manipulated based on the second position when the user's finger is 0 ohms away from the remote control device. In this way, the remote control device can support contact-based click operations.

[0020] In one possible design, after determining a second position on the display device based on the first position information and preset movement information corresponding to the first position information, the method further includes: when the height of the user's finger from the remote control device is less than or equal to a second height threshold, manipulating the object to be manipulated based on the second position, wherein the second height threshold is greater than 0 and less than or equal to a first height threshold. In this way, the remote control device can also support hover-type click operations.

[0021] In one possible design, the object to be manipulated, determined based on the second position, is a first control. This improves the accuracy of manipulating controls on the display device.

[0022] In one possible design, the display device has multiple displays, and the object to be manipulated, determined based on the second position, is the first display among the multiple displays. This improves the accuracy of manipulating the displays included in the display device.

[0023] In one possible design, the method is performed by the remote control device or the display device.

[0024] In one possible design, the first location information is acquired by the remote control device, and the second location is determined by the display device.

[0025] In one possible design, the method further includes: determining the position of the object to be manipulated on the display device as the distance between the user's finger and the remote control device decreases from the first height to the third height, wherein the third height is the height at which manipulation of the object to be manipulated is triggered based on the second position.

[0026] In this way, as the user's finger continues to fall from the first height, the position of the display device can be locked at the position predicted at the first height, which can further improve the accuracy of controlling the display device through the remote control device.

[0027] In one possible design, before acquiring the first position information corresponding to the projection position of the user's finger on the remote control device, the method further includes:

[0028] It is determined that the user's finger is within the detection range of the remote control device.

[0029] In one possible design, after determining the second position on the display device based on the first position information and the preset movement information corresponding to the first position information, the method further includes:

[0030] The control displays the identification information of the first position represented by the first position information at the second position. For example, the identification information can be a virtual finger, a cursor, etc.

[0031] In a second aspect, a system is provided, comprising a remote control device and a display device, wherein the remote control device and the display device are connected in communication, and the remote control device is used to control the display device; the remote control device is used to acquire first position information corresponding to the projection position of the user's finger on the remote control device when the user's finger is at a first height from the remote control device; the remote control device or the display device is used to determine a second position on the display device based on the first position information and preset movement information corresponding to the first position information, wherein the second position is used to determine an object to be controlled on the display device; the display device is used to perform an operation based on the object to be controlled determined by the second position.

[0032] In one possible design, the remote control device is specifically used to acquire first position information corresponding to the projection position of the user's finger on the remote control device when the height of the user's finger from the remote control device is the first height threshold, when the first height is less than or equal to the first height threshold.

[0033] In one possible design, the remote control device or the display device is further configured to determine a third position corresponding to the projection position of the user's finger on the remote control device as the position of the object to be manipulated on the display device when the height of the user's finger from the remote control device is greater than or equal to the first height threshold.

[0034] In one possible design, the preset movement information includes at least one of a preset movement distance and a preset movement direction.

[0035] In one possible design, the remote control device is further configured to acquire second position information corresponding to the projection position of the user's finger on the remote control device when the user's finger is at a second height from the remote control device; the first position represented by the first position information and the second position represented by the second position information are located in different display areas on the display device, or the user operation corresponding to the first position information is different from the user operation corresponding to the second position information, or the finger used by the user when acquiring the first position information is different from the finger used by the user when acquiring the second position information; the preset movement information corresponding to the first position information is different from the preset movement information corresponding to the second position information.

[0036] In one possible design, the remote control device is further configured to control the object to be controlled based on the second position when the user's finger is 0 liters away from the remote control device.

[0037] In one possible design, the remote control device is further configured to manipulate the object to be manipulated based on the second position when the height of the user's finger from the remote control device is less than or equal to a second height threshold, wherein the second height threshold is greater than 0 and less than or equal to a first height threshold.

[0038] In one possible design, the object to be manipulated, determined based on the second position, is a first control.

[0039] In one possible design, the display device has multiple displays, and the object to be manipulated, determined based on the second position, is the first display among the multiple displays.

[0040] In one possible design, the remote control device is further configured to determine that the position of the object to be manipulated on the display device is the second position as the distance between the user's finger and the remote control device decreases from the first height to the third height, wherein the third height is the height at which manipulation of the object to be manipulated is triggered based on the second position.

[0041] Thirdly, an apparatus is provided that has the function of implementing the method as described in any of the designs in the first aspect above. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described function.

[0042] Fourthly, a remote control device is provided, comprising: a processor, a communication interface, and a memory, wherein the memory and the communication interface are coupled to the processor, the communication interface is used to communicate with other devices, and the memory is used to store computer program code, the computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the remote control device causes the remote control device to perform the method as described in any of the designs in the first aspect above.

[0043] Fifthly, a display device is provided, comprising: a processor, a communication interface, a display screen, and a memory, wherein the memory, the communication interface, and the display screen are coupled to the processor, the communication interface is used for communicating with other devices, and the memory is used for storing computer program code, the computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the display device causes the display device to perform the method as described in any of the designs in the first aspect above.

[0044] Optionally, the memory can be coupled to the processor or independent of it. For example, the communication interface can be a transceiver, input / output interface, interface circuit, output circuit, input circuit, pins, or related circuitry. The display screen can be used by the display device to perform display operations.

[0045] A sixth aspect provides a computer-readable storage medium comprising a computer program that, when executed on a remote control device or a display device, causes the remote control device or display device to perform the method as described in any of the designs of the first aspect above.

[0046] In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform the method as described in any of the designs in the first aspect above.

[0047] Eighthly, a chip system is provided, including at least one processor and at least one interface circuit, the at least one interface circuit being used to perform transceiver functions and send instructions to at least one processor, wherein when at least one processor executes instructions, at least one processor performs the method as described in any of the designs in the first aspect above.

[0048] A ninth aspect provides a system including a remote control device and a display device, the remote control device being configured to perform a method performed by the remote control device as described in any of the preceding aspects, and the display device being configured to perform a method performed by the display device as described in any of the preceding aspects and any of the preceding aspects.

[0049] The technical effects of the aforementioned aspects can be referenced from each other, and will not be elaborated further here. Attached Figure Description

[0050] Figure 1 is a schematic diagram of a scenario where a display device is remotely controlled by a remote control device according to an embodiment of this application;

[0051] Figure 2 is a schematic diagram of another scenario where a display device is remotely controlled by a remote control device, as provided in an embodiment of this application.

[0052] Figure 3 is a schematic diagram of the architecture of a communication system provided in an embodiment of this application;

[0053] Figure 4 is a schematic diagram of the structure of a remote control device provided in an embodiment of this application;

[0054] Figure 5 is a structural schematic diagram of another remote control device provided in an embodiment of this application;

[0055] Figure 6 is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0056] Figure 7 is a schematic diagram of the detection range of a remote control device provided in an embodiment of this application;

[0057] Figure 8 is a schematic diagram of a control on a display device provided in an embodiment of this application;

[0058] Figure 9 is a schematic diagram of remote control devices and display devices of different size ratios provided in the embodiments of this application;

[0059] Figure 10 is a structural schematic diagram of another display device provided in an embodiment of this application;

[0060] Figure 11 is a schematic diagram of a process of presenting a virtual finger on a display device according to an embodiment of this application;

[0061] Figure 12 is a schematic diagram of another scenario where a display device is remotely controlled by a remote control device according to an embodiment of this application;

[0062] Figure 13 is a schematic diagram of a process for obtaining AI model training data according to an embodiment of this application;

[0063] Figure 14 is a schematic diagram of the data range of the landing position corresponding to the dot position provided in an embodiment of this application;

[0064] Figure 15 is a schematic diagram of a successful manipulation of the dot position on a display device using an AI model, provided by an embodiment of this application;

[0065] Figure 16 is a schematic diagram of another scenario where a display device is remotely controlled by a remote control device according to an embodiment of this application;

[0066] Figure 17 is a schematic diagram of a scenario for controlling the display screen of a display device according to an embodiment of this application;

[0067] Figure 18 is a schematic diagram of another scenario for controlling the display screen of a display device according to an embodiment of this application;

[0068] Figure 19 is a schematic diagram of a coordinate system provided in an embodiment of this application;

[0069] Figure 20 is a flowchart illustrating a device control method provided in an embodiment of this application;

[0070] Figure 21 is a flowchart illustrating another device control method provided in an embodiment of this application;

[0071] Figure 22 is a flowchart illustrating another device control method provided in an embodiment of this application;

[0072] Figure 23 is a structural schematic diagram of another remote control device provided in an embodiment of this application;

[0073] Figure 24 is a schematic diagram of the structure of another display device provided in an embodiment of this application;

[0074] Figure 25 is a schematic diagram of the structure of a chip system provided in an embodiment of this application. Detailed Implementation

[0075] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0076] In the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a and b and c, where a, b, and c can be single or multiple.

[0077] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0078] Currently, display devices can be remotely controlled via remote control devices (also known as remote terminals or control devices). These remote control devices support both touch-based and hover-based operations, with hover-based operations also known as hover touch control. When a user's finger hovers over the remote control device, the device can detect the finger signal using technologies such as capacitive sensing. This finger signal can then be transmitted and converted into a signal on the display device, enabling remote control of the device. For example, when a user's finger hovers and moves above the remote control device, the display device projects a virtual finger onto the screen based on this hovering finger. This virtual finger follows the user's finger on the display device, and the trajectory of the user's finger above the remote control device and the trajectory of the virtual finger on the display device are mapped in real time. Thus, in scenarios where the user's hand and eye are separated for remote control of the display device, the trajectory of the user's finger on the remote control device can be determined based on the trajectory of the virtual finger on the display device.

[0079] For example, Figure 1 illustrates a scenario of remotely controlling a display device via a remote control device according to an embodiment of this application. As shown in Figure 1, a user holds a remote control device 10 to remotely control a display device 20. When the movement trajectory of the user's finger above the remote control device 10 is trajectory 1, the movement trajectory of the virtual finger mapped on the display device 20 can be as trajectory 2.

[0080] When a user hovers their finger over a remote control to select a control on a display device, the finger falls over the remote to perform a click operation, thus clicking the selected control. However, during this descent, the user's finger may bend (or hook back), causing a displacement of its projected position on the remote. Consequently, the virtual finger mapped onto the display device also moves, resulting in the final controlled control not being the initially selected one, leading to lower accuracy.

[0081] For example, as shown in Figure 2, when the user's finger hovers above the remote control device 10 at a height of h1, the finger is not yet bent, and its projection on the device 10 is at position A1. Based on this, the virtual finger mapped onto the display device 20 is positioned at control A2, indicating that the user has selected control A2 and intends to manipulate it. Subsequently, the user's finger begins to fall from above the remote control device 10 to perform a click operation, thus controlling control A2. However, during this descent, the finger may bend. If the finger falls to a height of h2 above the device 10, its projection on the device 10 is now at position B1, which is offset relative to A1. Consequently, the position of the virtual finger on the display device 20 also shifts. For instance, if the virtual finger moves from control A2 to control B2, the final controlled control will not be A2.

[0082] Based on this, this application provides a device control method. When a user's finger falls to a certain height, instead of controlling the virtual finger on the display device to follow the user's finger, the method controls the virtual finger on the display device to move towards a predetermined position, thereby controlling the predetermined position. This can improve the accuracy of controlling the display device via a remote control.

[0083] For example, Figure 3 shows a schematic diagram of the architecture of a communication system for a device control method provided in an embodiment of this application. As shown in Figure 3, the communication system 300 includes a remote control device 301 and a display device 302.

[0084] The remote control device 301 can be used to receive user operations and send control signals to the display device 302, thereby controlling the display device 302. Optionally, the remote control device 301 can be a device dedicated to controlling the display device 302, or it can be a device with integrated control functions.

[0085] For example, the remote control device 301 may include, but is not limited to, a mouse as shown in Figure 4(1), a keyboard as shown in Figure 4(2) (optionally, the keyboard may be a physical keyboard or a virtual keyboard, a conventional large keyboard including multiple keys such as number keys and letter keys, or a small keyboard that supports user-defined keys, etc.), a laptop computer as shown in Figure 4(3), a touchpad as shown in Figure 4(4), a remote control, a smartphone, a laptop computer, a tablet computer, a handheld computer, a wearable device, an in-vehicle computer, a personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), or a game controller in a human-computer interaction scenario, etc. The remote control device 301 may also be other types or structures of devices with control functions and capable of receiving user operations. The remote control device 301 may be a portable device or a fixed device.

[0086] Optionally, in this embodiment, the user operation received by the remote control device 301 can be a contact operation or a hover touch operation. In some embodiments, the remote control device 301 may be configured with a touch area. When a user's finger hovers over the touch area, the remote control device 301 can detect the user's finger signal to achieve hover touch. Optionally, the touch area may also support the user performing contact operations. The touch area configured on the remote control device 301 may be as shown in Figure 4. Optionally, the touch area may be a touchpad or a touch screen with display function, etc.

[0087] Display device 302 can be any device with a display screen. Display device 302 can be used to display various controls (such as icons, text boxes, text, windows, cursors, search bars, etc.). Optionally, in this embodiment, the interface displayed on display device 302 can be displayed independently by display device 302, or it can be projected from other devices (such as, but not limited to, remote control device 301). Optionally, the display screen included in display device 302 can be a single independent display screen, such as a smart screen or computer display screen. It can also be a display screen array composed of one or more display screens, such as a monitoring display screen or a conference display screen.

[0088] For example, display device 302 may include, but is not limited to, televisions, smart screens, computers, in-vehicle systems, tablets, foldable screen devices, central control panels, projection devices, motion-sensing game consoles in human-computer interaction scenarios, and displays in various scenarios (such as conference screens, advertising screens, and traffic screens). Optionally, remote control device 301 and display device 302 may be the same type of device or different types of devices.

[0089] Optionally, the operating system installed on the aforementioned remote control device 301 and / or display device 302 includes, but is not limited to, Alternatively, other operating systems may be installed. Of course, the remote control device 301 and / or the display device 302 may not have an operating system installed. This application does not limit the specific type of the remote control device 301 and / or the display device 302, whether or not an operating system is installed, or the operating system installed if an operating system is installed.

[0090] Optionally, the remote control device 301 and the display device 302 can establish a connection via wired or wireless communication technology. For example, wireless communication technologies may include, but are not limited to, Bluetooth (BT) (e.g., classic Bluetooth or Bluetooth Low Energy (BLE) Bluetooth), Sparklink (e.g., classic Sparklink or Sparklink Low Energy (SLE)), Near Field Communication (NFC), Wireless Local Area Networks (WLAN) (e.g., Wireless Fidelity (Wi-Fi) networks), Zigbee, Frequency Modulation (FM), Infrared (IR), etc.

[0091] For example, Figure 5 shows a schematic diagram of the structure of a remote control device 301 provided in an embodiment of this application. As shown in Figure 5, the remote control device 301 includes a processor 510, a memory 520, a power module 530, a communication module 540, a sensor module 550, and buttons 560.

[0092] The processor 510 may include one or more processing units, such as a modem processor, a controller, a digital signal processor (DSP), a baseband processor, etc. These different processing units may be independent devices or integrated into one or more processors.

[0093] The memory 520 can be used to store computer executable program code, which includes instructions. The memory 520 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the remote control device 301, etc. Furthermore, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the remote control device 301 by running instructions stored in the memory 520 and / or instructions stored in memory disposed within the processor.

[0094] In some embodiments of this application, the memory 520 may store an AI model. This AI model can be used to predict the position of a virtual finger on the display device 302. A description of this AI model is provided below.

[0095] The power module 530 can be used to connect to a power supply to power the processor 510, memory 520, communication module 540, sensor module 550, etc.

[0096] The communication module 540 may include at least one of a wired communication module and a wireless communication module. The wired communication module can be used to implement the wired communication function of the remote control device 301. The wireless communication module can be used to implement the wireless communication function of the remote control device 301.

[0097] The wireless communication module can provide solutions for wireless communication applications on the remote control device 301, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Starflash, global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0098] In some embodiments, the wireless communication module may be coupled to an antenna, enabling the remote control device 301 to communicate with a network and other devices (such as, but not limited to, the display device 302) via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, Starflash, GNSS, WLAN, NFC, FM, and / or IR technologies, etc.

[0099] The sensor module 550 can be used to sense the user's finger signal and identify the position of the user's finger hovering above the remote control device 301 to achieve hover touch control. Alternatively, the sensor module 550 can also be used to identify the position of the user's finger touching the remote control device 301 to achieve contact operation. In some embodiments, the sensor module 550 may include a capacitive sensor. Optionally, the sensor module 550 can be disposed in a touch area configured on the remote control device 301, so that when the user's finger hovers above or touches the touch area, the sensor module 550 can sense the user's finger signal.

[0100] Button 560 can be a mechanical button or a touch button. The remote control device 301 can receive button input and generate key signal inputs related to user settings and function control of the remote control device 301. In some embodiments of this application, button 560 can be used by the remote control device 301 to receive contact-based operations performed by the user, such as, but not limited to, contact-based click operations.

[0101] For example, FIG6 shows a schematic diagram of the structure of a display device 302 provided in an embodiment of this application. As shown in FIG6, the display device 302 includes a processor 610, a memory 620, a universal serial bus (USB) interface 630, a charging management module 640, a power management module 641, a battery 642, a communication module 650, a display screen 660, etc.

[0102] Processor 610 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0103] The processor 610 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 610 is a cache memory. This memory can store instructions or data that the processor 610 has just used or that are used repeatedly. If the processor 610 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 610, and thus improves the efficiency of the system.

[0104] USB interface 630 is an interface that conforms to the USB standard specification. USB interface 630 can be used to connect a charger to charge display device 302, and can also be used for data transfer between display device 302 and peripheral devices.

[0105] The charging management module 640 is used to receive charging input from the charger.

[0106] The power management module 641 is used to connect the battery 642, the charging management module 640, and the processor 610. The power management module 641 receives input from the battery 642 and / or the charging management module 640 to power the processor 610, memory 620, and communication module 650, etc.

[0107] The display screen 660 is used to display images, videos, etc. The display screen 660 includes a display panel. In some embodiments, the display device 302 may include one or N display screens 660, where N is a positive integer greater than 1. In some embodiments of this application, the display screen 660 can be used to present various controls and screens. In still other embodiments of this application, the display screen 660 can also be used to present mapped virtual fingers.

[0108] For a description of the memory 620 and communication module 650 shown in Figure 6, please refer to the description of the memory 520 and communication module 540 shown in Figure 5.

[0109] It is understood that the structures illustrated in Figures 5 and 6 do not constitute a specific limitation on the remote control device 301 and the display device 302. In other embodiments of this application, the remote control device 301 or the display device 302 may include more or fewer components than illustrated. For example, the remote control device 301 may also include a display screen for performing display operations. As another example, the display device 302 may also include an audio module for audio output. Alternatively, the remote control device 301 or the display device 302 may combine some components, separate some components, or have different component arrangements. The processing steps or functional characteristics of the illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0110] The technical solutions involved in the following embodiments can all be implemented in devices with the structures shown in Figures 5 and 6, and in systems with the architecture shown in Figure 3.

[0111] In some scenarios, when a user controls a display device via a remote control, the user can hover their finger above the remote control. The remote control can then detect the user's finger signal.

[0112] In this scenario, in some embodiments, the detection range of the remote control device is limited. The remote control device can only detect the user's finger signal when the user's finger is within its detection range. Conversely, when the user's finger is outside the detection range, the remote control device cannot detect the user's finger signal. For example, the detection range of the remote control device can be a space 700 as shown in Figure 7, with a height of H, a width of W, and a length of L. The height H refers to the maximum height at which the remote control device can detect the user's finger signal, the width W refers to the maximum width at which the remote control device can detect the user's finger signal, and the length L refers to the maximum length at which the remote control device can detect the user's finger signal. Based on the detection range shown in Figure 7, the user's finger being within the detection range of the remote control device means being within the space 700 shown in Figure 7.

[0113] Furthermore, the remote control device can operate the display device based on detected finger signals. In some embodiments, the remote control device can operate any control presented on the display device based on detected finger signals. For example, in the game scene shown in FIG8 (1), the controls presented on the display device may include, but are not limited to, orientation controls 801 to 804, game item controls 805 to 807, etc. For another example, in the video playback scene shown in FIG8 (2), the controls presented on the display device may include, but are not limited to, the video progress bar control 810, etc. It is understood that FIG8 only illustrates examples of a few controls, and the embodiments of this application are also applicable to other types of controls, such as windows in split-screen scenarios.

[0114] Optionally, in this embodiment, the display device may include one display screen. This embodiment does not limit the size ratio between the remote control device and the display device, that is, it does not limit the size ratio between the touch area of ​​the remote control device and the display screen of the display device. For example, Figures 9(1) to 9(4) show examples of several different size ratios provided in the embodiments of this application.

[0115] In other embodiments, when the display device is a display array composed of multiple displays spliced ​​together, the remote control device can also control different displays of the display device based on detected finger signals, such as magnifying or brightening the screen. For example, Figures 10(1) and 10(2) show schematic diagrams of the display device provided in the embodiments of this application. As shown in Figures 10(1) and 10(2), the display device is a display array composed of 12 displays spliced ​​together.

[0116] Optionally, in this embodiment, when the image of a display screen is magnified to be displayed using a display array, the remote control device can also operate the controls in the image based on the detected finger signals.

[0117] In some embodiments, when the remote control device detects a finger signal floating above it, it can determine the position of the mapped virtual finger on the display device based on the position of the finger signal projected on the remote control device, so as to present the virtual finger on the display device. For example, as shown in Figure 11(1), taking a user's finger floating above the remote control device at a height of h1, the remote control device can determine the position of the user's finger projected on the remote control device as A1 based on the user's finger signal. Optionally, when the user's finger is floating above the remote control device at a height of h1, the user's finger may not be bent downwards.

[0118] Then, the remote control device can determine the coordinates (x1, y1) of position A1 in the coordinate system of the remote control device. As shown in Figure 11(1), the coordinate system of the remote control device can be a coordinate system established with reference to the touch area. Next, the remote control device can determine the coordinates (x2, y2) of position A2 mapped from position A1 in the coordinate system of the display device based on the size ratio between the remote control device and the display device, and send the coordinates of position A2 to the display device. As shown in Figure 11(1), the coordinate system of the display device can be a coordinate system established with reference to the display screen of the display device. Then, the display device can present a virtual finger at position A2 to facilitate the user in determining the currently selected area or position on the display device.

[0119] It is understood that the coordinate system shown in Figure 11 is merely an illustrative example, and other settings may be used in actual applications. It is also understood that in this embodiment, the virtual finger displayed on the display device is merely an indicator to facilitate the user's observation of the currently selected or operated position on the display device. The virtual finger can be presented in the shape of a finger or other physical forms, such as a cursor. In this embodiment, it is presented as a finger.

[0120] Optionally, when the user's finger hovers above the remote control device, the user's finger may also move horizontally. The horizontal direction can refer to the direction parallel to the touch area of ​​the remote control device. For example, the user's finger may gradually move horizontally from the position shown in Figure 11(1) to the position shown in Figure 11(2). Correspondingly, the remote control device can also determine the real-time position projected on the remote control device based on the moving user's finger, which can be from position A1 to position B1 on the remote control device as shown in Figure 11(2). The remote control device can determine the real-time position of the virtual finger on the display device based on the real-time position projected on the remote control device of the moving user's finger and the size ratio between the remote control device and the display device, which can be from position A2 to position B2 on the display device as shown in Figure 11(2). This allows the virtual finger displayed on the display device to follow the movement of the user's finger.

[0121] The above description uses the example of the remote control device directly determining the position of the virtual finger on the display device and sending that position to the display device. In other implementations, the remote control device can also directly send the position of the user's finger projected on the remote control device to the display device, which then determines the position of the virtual finger on the display device based on the position of the user's finger projected on the remote control device and the size ratio between the remote control device and the display device.

[0122] The above describes how to implement virtual fingers on a display device. The following example illustrates the process of a remote control manipulating controls on a display device based on detected finger signals.

[0123] In some embodiments, a user can perform a click operation on the remote control device. In response to this operation, the remote control device can manipulate controls on the display device. When the user performs the click operation, their finger, hovering above the remote control device, gradually descends until it contacts the device. During this descent, the remote control device can detect the user's finger signal in real time to determine the distance between the finger and the device. Based on this distance, it then determines which control on the display device will be manipulated.

[0124] In this embodiment, as the user's finger falls above the remote control device, when the distance between the user's finger and the remote control device is greater than or equal to a height threshold of 1, the virtual finger on the display device can follow the user's finger. The control displayed at the location of the virtual finger is the selected control, i.e., the control to be manipulated. Specifically, referring to Figure 11, the remote control device can determine the position of the user's finger projected on the remote control device based on the position of the user's finger hovering above the remote control device, and then determine the position of the mapped virtual finger on the display device based on the position of the user's finger projected on the remote control device and the size ratio between the remote control device and the display device. At this time, the control displayed at the location of the virtual finger is the selected control.

[0125] Next, the user's finger continues to fall above the remote control device. When the height between the user's finger and the remote control device is less than or equal to a height threshold of 1, unlike the method described above for determining the selected control, the remote control device can predict the selected control. Specifically, when the remote control device detects that the height between the user's finger and the remote control device is the aforementioned height threshold of 1, the remote control device can first determine the position of the mapped virtual finger on the display device based on the current position of the user's finger hovering above the remote control device. Similarly, the remote control device can also determine the position of the user's finger projected onto the remote control device based on the current position of the user's finger hovering above the remote control device, and then determine the position of the mapped virtual finger on the display device based on the position of the user's finger projected onto the remote control device and the size ratio between the remote control device and the display device.

[0126] Furthermore, the remote control device can determine the position of the virtual finger after it has moved based on the position of the mapped virtual finger on the display device. At this point, the control displayed at the location of the moved virtual finger is the selected control. Specifically, the remote control device can determine the direction and distance the virtual finger needs to move on the display device based on the position of the mapped virtual finger on the display device, and then determine the position of the virtual finger after it has moved based on this direction and distance. This position is the predicted position of the virtual finger.

[0127] Optionally, in this embodiment of the application, taking the use of a capacitive sensor to sense the user's finger signal in the remote control device as an example, the remote control device can determine the height of the user's finger from the remote control device and the position of the user's finger projected on the remote control device based on the collected capacitance value.

[0128] The above process is described in conjunction with Figure 12. As shown in Figure 12(1), initially, the user's finger is suspended above the remote control device at a height of h1, which is greater than the height threshold 1 (i.e., hs). The remote control device determines the position of the user's finger projection on the remote control device as A1 based on the finger signal at this time. And as shown in Figure 12(2), the remote control device determines the position A2 of the mapped virtual finger on the display device based on position A1. Then, as shown in Figure 12(3), the user's finger begins to fall from above the remote control device until it falls to a height of hs. During the process of the user's finger falling from height h1 to height hs, the remote control device can sense the user's finger signal in real time and determine the real-time position of the user's finger projection on the remote control device based on the sensed finger signal, and then determine the real-time position of the mapped virtual finger on the display device based on the real-time position of the projection. As shown in Figure 12(3), the position of the user's finger projection on the remote control device changes continuously from position A1 to position B1. As shown in Figure 12(4), the virtual finger moves continuously from position A2 to position B2 on the display device. During this process, the virtual finger follows the user's finger as it moves on the display device. Control B2 is displayed at position B2.

[0129] When the user's finger is at a height of hs from the remote control device, as shown in Figure 12(3), the position of the user's finger projected on the remote control device is position B1. As shown in Figure 12(4), the position of the mapped virtual finger on the display device is position B2. Then, as shown in Figure 12(5), the user's finger continues to fall above the remote control device. At this time, the height of the user's finger from the remote control device is h2, which is less than hs. At this time, the position of the user's finger projected on the remote control device is position C1. The remote control device can determine the direction that the virtual finger needs to move on the display device as direction a1 and the distance that needs to move as distance b1 based on the position of the mapped virtual finger on the display device when the user's finger is at a height of hs (i.e., position B2 shown in Figure 12(4)). Then, as shown in Figure 12(6), the virtual finger displayed on the display device changes from position B2 to position C2. At this time, position C2 is no longer mapped from position C1 and is unrelated to position C1. Control C2 is displayed at position C2. At this time, the selected control is control C2. Finally, as shown in Figure 12 (7), the user's finger continues to fall above the remote control device until it touches the remote control device, that is, the distance from the remote control device is 0. In response to this operation, the remote control device can control the control C2 on the display device.

[0130] In this embodiment, when a user's finger hovers above the remote control device and the distance between the finger and the remote control device is less than or equal to a height threshold of 1, it indicates that the user may be performing a click operation on the remote control device. Therefore, instead of controlling the virtual finger on the display device to follow the user's finger, the system anticipates the control to be manipulated, improving the accuracy of controlling the display device via the remote control. Optionally, in this embodiment, the height threshold of 1 can be set based on empirical values.

[0131] In this embodiment, as one possible implementation, the remote control device may have a pre-installed artificial intelligence (AI) model. Optionally, this AI model can be various machine learning models or deep learning models. The remote control device can input the position of the virtual finger mapped when the user's finger is at a height threshold of 1 into the AI ​​model, and the AI ​​model will output the direction and distance that the virtual finger needs to move on the display device.

[0132] The process of obtaining the AI ​​model is described below.

[0133] In some embodiments, as shown in Figure 13, the display screen of the display device can be divided into multiple display areas, and the position of each display area can be referred to as a dot position. Then, a click operation can be performed on the remote control device to manipulate the dot positions divided on the display device. For each dot position, multiple click operations can be performed, and the actual mapped position point on the display device for each click operation is collected. An actual mapped position point can be referred to as a landing point, and the position of the landing point can be simply referred to as the landing point position. The implementation of determining the actual mapped position on the display device based on the click operation performed on the remote control device can refer to the implementation of determining the position of the mapped virtual finger on the display device based on the position of the user's finger projected on the remote control device described above.

[0134] The actual position controlled by the click operation may differ from the desired click position. Each click position may have multiple corresponding landing positions mapped onto the display device based on the click operation. For example, as shown in Figure 13(1), taking positions A, B, C, and D on the display device as the desired click positions, the click positions corresponding to multiple click operations performed on the remote control device can be shown in Figure 13(2), which only shows a portion. The landing positions on the display device mapped from these positions on the remote control device can be shown in Figure 13(1). Each click position in positions A, B, C, and D has multiple landing positions around it. Based on the landing positions shown in Figure 13(1), for example, Figures 14(1) to 14(4) show the data range of the landing positions corresponding to each click position in positions A, B, C, and D, respectively. This data range can be the range enclosed by the ellipse shown in Figures 14(1) to 14(4). In order to control the position of the data point, all points within the data range need to be moved toward the target point.

[0135] Then, the aforementioned click positions and their corresponding landing positions are input into the model for training. This yields the direction and distance required to reach each landing position, thus creating the aforementioned AI model. For example, Figure 15 illustrates the successful manipulation of positions A, B, C, and D on the display device using this AI model. As shown in Figure 15, the landing positions determined by the user's click operation on the remote control device coincide with positions A, B, C, and D, respectively, enabling successful clicking on these positions.

[0136] As another possible implementation, the remote control device can also store a mapping relationship between the position of the virtual finger on the display device and the direction and distance that the virtual finger needs to move on the display device. The remote control device can determine the direction and distance that the virtual finger needs to move on the display device based on the mapped position of the virtual finger on the display device and the stored mapping relationship.

[0137] Optionally, for the same remote control device, the direction and distance the virtual finger needs to move on the display device may differ due to the varying scales between the remote control device and different sized display devices. Therefore, the pre-set AI model or mapping relationship in the remote control device can be specific to the currently controlled display device. Alternatively, the remote control device can also have pre-set AI models or mapping relationships corresponding to different sized display devices, allowing it to use the AI ​​model or mapping relationship corresponding to the currently controlled display device to determine the direction and distance the virtual finger needs to move on that device. Furthermore, the remote control device can input the size ratio between itself and the currently controlled display device into the AI ​​model, which then outputs the direction and distance the virtual finger needs to move on that device. Optionally, the AI ​​model can also be trained by additionally considering the size ratio between the remote control device and the display device. Alternatively, the mapping relationship can also be a mapping relationship between the size ratio between the remote control device and the display device, the position of the virtual finger on the display device, and the direction and distance that the virtual finger needs to move on the display device. Based on this mapping relationship, the remote control device can also determine the direction and distance that the virtual finger needs to move on the display device, in combination with the size ratio between the remote control device and the display device, and the position of the virtual finger on the display device based on the mapping.

[0138] In this embodiment, the example used is a remote control device determining the direction and distance a virtual finger needs to move on a display device. In other embodiments, the display device can also perform this operation. For example, the display device may have the aforementioned AI model and mapping relationship pre-installed. The remote control device can send the detected height of the user's finger from the remote control device and the position of the user's finger projected on the remote control device to the display device, which then determines which control on the display device to manipulate. The specific implementation can be referenced from the implementation of the remote control device.

[0139] Optionally, in the above embodiments, when the user's finger rises from above the remote control device, the virtual finger displayed on the display device can follow the movement of the user's finger. That is, during this process, regardless of whether the height of the user's finger from the remote control device is less than or equal to the height threshold 1, the remote control device can determine the position of the user's finger projected on the remote control device based on the position of the user's finger hovering above the remote control device, and then determine the position of the mapped virtual finger on the display device based on the position of the user's finger projected on the remote control device and the size ratio between the remote control device and the display device. For example, referring to the example shown in Figure 12, as shown in Figure 12 (1), after the user finishes the click operation, the user's finger starts to rise from above the remote control device. During the process of the user's finger rising from a position with a height of 0 from the remote control device to a height of hs (i.e., height threshold 1) as shown in Figure 12 (3), the remote control device can sense the user's finger signal in real time, and determine the position of the user's finger projected on the remote control device based on the sensed finger signal, and then determine the position of the mapped virtual finger on the display device based on the projected position.

[0140] Assuming the user's finger is at a height of 0 meters from the remote control device, its projected position on the remote control device is position D1 (not shown in Figure 12), and the user's finger is at a height of hs, its projected position on the remote control device is position B1. Then, as the user's finger rises from position 0 meters to height hs, the projected position of the user's finger on the remote control device continuously changes from position D1 to position B1. Correspondingly, assuming the virtual finger mapped from position D1 is at position E1 on the display device, and the virtual finger mapped from position B1 is at position B2 on the display device, then the virtual finger on the display device follows the user's finger, continuously moving from position E1 to position B2.

[0141] Furthermore, as the user's finger continues to rise above the remote control device, such as from height hs as shown in Figure 12 (3) to height h1 as shown in Figure 12 (1), the remote control device can continue to sense the user's finger signal in real time, and determine the position of the user's finger projection on the remote control device based on the sensed finger signal. Then, based on the position of the projection, it determines the position of the mapped virtual finger on the display device. The position of the virtual finger on the display device will continue to move following the change in the position of the user's finger projection.

[0142] The above embodiments use a contact-based click operation as an example. In other embodiments, the click operation can also be a hovering click operation, i.e., a click operation that does not contact the remote control device. In this embodiment, as the user's finger falls above the remote control device, when the remote control device detects that the distance between the user's finger and the remote control device is less than or equal to a height threshold 2, it determines that the user has performed a click operation, and then performs manipulation on the selected control. The height threshold 2 can be less than the height threshold 1.

[0143] In this embodiment, referring to the process shown in Figure 12, when the user's finger continues to fall from above the remote control device to a height h2 as shown in Figure 12 (5), after the remote control device determines that the control to be manipulated is control C2, the remote control device does not need to manipulate control C2 on the display device when it detects that the height of the user's finger from the remote control device is 0. Instead, as shown in Figure 16, the remote control device can manipulate control C on the display device when it detects that the height of the user's finger from the remote control device is less than or equal to the height threshold 2 (i.e., hm). When the height of the user's finger from the remote control device is hm, the position of the user's finger projected on the remote control device can be as shown in position D1. For a description of the other parameters shown in Figure 16, please refer to the corresponding description in Figure 12.

[0144] Optionally, in this embodiment, as the user's finger rises from above the remote control device, the remote control device can also sense the user's finger signal in real time. If the height of the user's finger from the remote control device is less than height threshold 1 but greater than height threshold 2, the remote control device can also determine the direction and distance the virtual finger needs to move on the display device using the method described above. That is, during this process, the virtual finger on the display device may not follow the user's finger. This improves the accuracy of controlling the display device using non-contact click operations.

[0145] The above embodiments are described using the example of a remote control device manipulating controls on a display device. Similarly, the scheme described in the above embodiments is also applicable to the scheme of manipulating the display screen of a display device. For example, taking the display device shown in Figure 10 (1) as an example, combined with the process shown in Figure 12, as shown in Figure 12 (3), when the user's finger falls from above the remote control device to a height of hs (i.e., height threshold 1) from the remote control device, the position of the user's finger projected on the remote control device is position B1. As shown in Figure 17 (1), the virtual finger is displayed at the position of display screen 1, and display screen 1 is selected at this time. Subsequently, as shown in Figure 12 (5), the user's finger continues to fall above the remote control device, and the height of the user's finger from the remote control device is h2, where h2 is less than hs. At this time, the remote control device determines the direction and distance that the virtual finger needs to move based on the position of the virtual finger on the display device when the height of the user's finger from the remote control device is hs. As shown in Figure 17 (2), the virtual finger on the display device changes from the position of display screen 1 to the position of display screen 2. At this time, display screen 2 is selected. Finally, as shown in Figure 12 (7), the user's finger continues to fall above the remote control device until it touches the remote control device, i.e., the distance from the remote control device is 0. In response to this operation, the remote control device can control the display screen 2 on the display device. As shown in Figure 17 (3), the image displayed on the display screen 2 is magnified and displayed on all the display screens included in the display device.

[0146] Similarly, taking the display device shown in Figure 10 (2) as an example, combined with the process shown in Figure 12, as shown in Figure 12 (3), when the user's finger falls from above the remote control device to a height of hs (i.e., height threshold 1) above the remote control device, the position of the user's finger projected on the remote control device is position B1. As shown in Figure 18 (1), the virtual finger is displayed at the position of display screen 3, and display screen 3 is selected at this time. Subsequently, as shown in Figure 12 (5), the user's finger continues to fall above the remote control device, and the height of the user's finger from the remote control device is h2, which is less than hs. At this time, the remote control device determines the direction and distance that the virtual finger needs to move based on the position of the virtual finger on the display device when the height of the user's finger from the remote control device is hs. As shown in Figure 18 (2), the virtual finger on the display device changes from the position of display screen 3 to the position of display screen 4. Display screen 4 is selected at this time. Finally, as shown in Figure 12 (7), the user's finger continues to fall above the remote control device until it contacts the remote control device, i.e., the height from the remote control device is 0. In response to this operation, the remote control device can control the display screen 4 on the display device. As shown in Figure 18(3), the image displayed on display screen 4 is magnified and displayed on all the display screens included in the display device.

[0147] In some embodiments, in addition to predicting the position of the virtual finger after movement, pre-configured locking conditions can also be used. During the movement of the user's finger while it hovers within a detection range such as that shown in Figure 7, if the remote control device determines that the current movement state of the user's finger meets the locking conditions, it can lock the three-dimensional coordinates, that is, lock the position of the user's finger projected on the remote control device, thereby locking the display position of the virtual finger. In this embodiment, the three-dimensional coordinates can refer to the coordinates of the user's finger in the t-coordinate system determined by the remote control device based on the detected finger signal, which can be used to characterize the position of the user's finger hovering above the remote control device. Optionally, after locking the display position of the virtual finger, for example, it can continue to cooperate with the prediction of the virtual finger's display position on the screen, and move the finger to the predicted display position, thereby locking the display position of the virtual finger at the predicted display position. Optionally, the height threshold for triggering position prediction and the height threshold for triggering locking can be different or the same. For example, the height for triggering locking can be higher than the height for triggering prediction, that is, the projection position of the virtual finger is locked for a short period of time first, and the display position of the virtual finger is predicted and executed during the continued descent. For example, the trigger lock height can be the same as the trigger prediction height, meaning that position locking and position prediction are executed simultaneously to directly obtain the predicted position information. Optionally, after determining the predicted position information, if the user's finger is still in a hovering and descending state and has not yet touched the surface of the remote control device, position locking can continue to be executed to improve operational accuracy.

[0148] For example, as shown in Figure 19, the plane containing the touch area of ​​the remote control device is the plane containing the X and Y axes, and the direction perpendicular to this plane is the Z axis, creating a t-coordinate system. For example, the t-coordinate system's X-axis, Y-axis, and Z-axis are shown in Figure 19. Based on this t-coordinate system, the remote control device determines the three-dimensional coordinates (x, y, z) corresponding to the user's finger. The difference between this t-coordinate system and the coordinate system of the aforementioned remote control device is that the t-coordinate system includes a Z-axis, while the X and Y axes are set in the same way as in the aforementioned remote control device coordinate system.

[0149] Optionally, the locking conditions may include, for example, that the user's finger height is less than or equal to a height threshold of 1, and the user's finger's downward movement speed in the z-axis direction is greater than a z-axis movement speed threshold, its movement speed in the x-axis direction is less than an x-axis movement speed threshold, and its movement speed in the y-axis direction is less than a y-axis movement speed threshold. That is, if the user's finger shows a clear downward trend over multiple consecutive frames and no significant movement trend in the x and y axes, the locking conditions can be determined. Optionally, locking the three-dimensional coordinates means that after locking, the three-dimensional coordinates used to determine the virtual finger's display position no longer change with changes in the user's finger's corresponding three-dimensional coordinates. After the remote control device obtains the three-dimensional coordinates A, it determines that the locking conditions are met, and subsequently sends three-dimensional coordinates A to the display device. Thus, the virtual finger's display position determined by the display device based on the same three-dimensional coordinates A will also remain unchanged, achieving virtual finger locking. For another example, after the remote control device obtains the three-dimensional coordinates A, it determines that the locking conditions are met, and subsequently uses three-dimensional coordinates A in all three-dimensional coordinate transformations. Therefore, after the touch conditions are met, the display position sent by the remote control device to the display device is transformed based on the same three-dimensional coordinate A, thereby enabling the display device to display the virtual finger at the same display position and achieve virtual finger locking. That is, the embodiments of this application do not limit whether the three-dimensional coordinate transformation is performed by the remote control device or the display device.

[0150] It is understood that in the embodiments of this application, locking the three-dimensional coordinates means locking the two-dimensional coordinates (x, y) corresponding to the three-dimensional coordinates on the remote control device, which means locking the position of the user finger projected on the remote control device to determine the display position of the virtual finger, which means locking the position of the virtual finger on the display device.

[0151] Optionally, the display effect of the virtual finger on the display device may change as the user's finger height changes. For example, as the user's finger height decreases, the virtual finger displayed on the display device becomes darker and larger; as the user's finger height increases, the virtual finger displayed on the display device becomes lighter and smaller. Optionally, during the virtual finger locking process, the display position of the virtual finger needs to be locked, but the display effect of the virtual finger does not need to be locked. Therefore, during the virtual finger locking process, the remote control device locks the x-axis and y-axis coordinates in the three-dimensional coordinate system, but does not need to lock the z-axis coordinate. That is, locking the three-dimensional coordinates means that the remote control device or display device obtains the display position of the virtual finger based on the locked x-axis and y-axis coordinates, and obtains the display effect of the virtual finger based on the actual z-axis coordinate, i.e., the z-axis coordinate is not locked. So during the locking process, the display position will not change, and the display effect will change with the user's finger height. Alternatively, the three-dimensional coordinate locking can also lock all three axes: x-axis, y-axis, and z-axis.

[0152] Optionally, the remote control device can determine whether the locking conditions are met using the following formula group one.

[0153] Among them, z n Let z be the height of the three-dimensional coordinates obtained based on the user's finger position in the nth frame, and z be the height of the three-dimensional coordinates. thresh The height threshold is 1. vz n Let vz be the speed at which the user's finger moves along the z-axis in the nth frame. thresh This represents the threshold for the movement speed along the z-axis. n Let vx be the speed at which the user's finger moves along the x-axis in the nth frame. thresh Here, is the threshold for movement speed along the x-axis. vyn is the movement speed of the user's finger along the y-axis in the nth frame. thresh This represents the threshold for the y-axis movement speed. Here, n is a positive integer greater than or equal to 2.

[0154] Among them, the three-dimensional velocity (vx) corresponding to the three-dimensional coordinates of the user's finger. n ,vy n ,vz n vx in ) n =(x n -x n-1 ) / Δt, vy n =(y n -yn-1) / Δt,vz n =(z n -z n-1 ) / Δt.

[0155] Optionally, the remote control device can use a hover touch algorithm to track the user's finger position and obtain the three-dimensional coordinates and velocity of the user's finger. Optionally, the hover touch algorithm can be, for example, a Kalman filter algorithm. Kalman filtering is a recursive algorithm widely used to estimate the state of dynamic systems, especially when noise is present in the system. By combining the mathematical model of the system and noise information, Kalman filtering can optimally estimate unknown variables or states and continuously update the estimates based on new observation data over time.

[0156] Optionally, the above description uses changes in three-dimensional coordinates to determine whether the locking condition is met. It should be understood that this application does not limit the method used to determine the movement state of the user's finger. For example, the remote control device can also directly determine whether the locking condition is met based on changes in detected capacitance data. For example, the remote control device can determine the locking condition based on changes in the z-axis height in the three-dimensional coordinates and whether the descent speed has a significant decreasing trend. The remote control device determines the three-dimensional coordinates based on the detected capacitance data. Therefore, the remote control device can also directly determine whether the finger is sufficiently close to the touch area of ​​the remote control device and whether the descent speed has a significant decreasing trend based on changes in capacitance data. The following description still uses the example of the remote control device determining the movement state of the user's finger through changes in three-dimensional coordinates to achieve virtual finger locking or unlocking, to introduce the device control method provided in this application embodiment. It should be understood that the remote control device can also achieve virtual finger locking or unlocking through changes in capacitance data and other methods.

[0157] The aforementioned three-dimensional coordinate locking scheme allows the three-dimensional coordinates to be locked at the same position as the user's finger continues to fall, provided the locking conditions are met, after the virtual finger's display position has been predicted. This means the projected position of the user's finger on the remote control device is locked at the same location, thus keeping the virtual finger's display position locked at the predicted position. Therefore, by predicting the virtual finger's display position and then displaying it at the predicted position as the user's finger continues to fall, provided the locking conditions are met, the accuracy of controlling the display device via remote control can be further improved.

[0158] In some embodiments, during the virtual finger locking process, the remote control device continuously detects the three-dimensional coordinates of the user's finger. The remote control device can then determine the movement state of the user's finger based on changes in the three-dimensional coordinates, thereby determining whether the unlocking conditions are met. Optionally, the unlocking conditions may include one or more of the following: no touch event detected within a preset time, x-axis offset exceeding a lateral offset threshold, y-axis offset exceeding a longitudinal offset threshold, and detection of a lift-off event.

[0159] For example, the remote control device predicts the user's intention to tap the device's touch area based on detected 3D coordinates, and thus locks the detected 3D coordinates. If the remote control device does not detect a touch event within a preset time, it can be determined that the user's intention was incorrectly confirmed, thus meeting the unlocking condition. Therefore, the remote control device can unlock the 3D coordinates and send the latest detected real 3D coordinates to the display device, triggering the display device to display the corresponding virtual finger according to those real 3D coordinates. In this embodiment, the touch event can be generated when the user's finger touches the touch area of ​​the remote control device, or when the height of the user's finger from the remote control device is less than or equal to a height threshold of 2. Thus, by unlocking the virtual finger in a timely manner, it prevents the user from mistakenly locking the virtual finger when there is no intention to touch it, thus avoiding affecting the user's subsequent operations.

[0160] Optionally, the remote control device determines that the locking condition is met based on the three-dimensional coordinate data obtained in the m-th frame, and triggers the locking of the virtual finger. Then, the remote control device can determine the preset time based on data such as the z-axis height in the three-dimensional coordinates obtained in the m-th frame, the user's finger's movement speed in the z-axis direction, and the refresh rate of the remote control device's detection of the user's finger position.

[0161] For example, the remote control device can obtain the preset frame number k using the following formula. The remote control device acquires the user's finger position frame by frame according to the refresh rate, therefore the preset time can be defined by the preset frame number.

[0162] Among them, z m Let vz be the height of the z-axis in the 3D coordinate system obtained in the m-th frame. m fs represents the speed at which the user's finger moves along the z-axis in the m-th frame, and fs represents the refresh rate of the remote control device.

[0163] For example, if, before a touch event is detected within a preset time, the remote control device detects that the user's finger has moved beyond a certain range around the locked 3D coordinates in the x-axis and / or y-axis planes—for instance, the x-axis offset exceeds a horizontal offset threshold and / or the y-axis offset exceeds a vertical offset threshold—then the remote control device can determine that the user's intent confirmation is incorrect, thus meeting the unlocking conditions. Therefore, the remote control device can unlock the 3D coordinates and send the newly detected actual 3D coordinates to the display device, triggering the display device to display the corresponding virtual finger according to those actual 3D coordinates. In this way, by unlocking the virtual finger in a timely manner, problems caused by the user mistakenly locking the virtual finger when there is no intention to touch are avoided, preventing disruption to subsequent user operations.

[0164] For example, if the remote control detects a touch event within a preset time and confirms the user's intent is correct, it can continue locking the 3D coordinates. Subsequently, during the 3D coordinate locking process, if the remote control detects that the user's finger movement in the x-axis and / or y-axis planes exceeds a certain range around the locked 3D coordinates—for example, the x-axis offset exceeds a horizontal offset threshold and / or the y-axis offset exceeds a vertical offset threshold—then the remote control can determine that the user has an intention to slide, meeting the unlocking conditions. Therefore, the remote control can unlock the 3D coordinates and send the newly detected real 3D coordinates to the display device, thereby triggering the display device to display the corresponding virtual finger according to the real 3D coordinates. In this way, during the virtual finger locking process, the remote control can also unlock using the virtual finger, responding promptly to user operations and improving the user experience.

[0165] For example, if the remote control detects a touch event within a preset time and confirms that the user's intention is correct, it can continue to lock the 3D coordinates. Subsequently, during the 3D coordinate locking process, if the remote control detects a finger lift-off event corresponding to the user's finger, it can determine that the user's click operation has been completed and an instruction is needed to change the virtual finger state, such as from Touch state to Hover state, to meet the unlocking conditions. Therefore, the remote control unlocks the 3D coordinates and sends the latest detected real 3D coordinates to the display device, thereby triggering the display device to display the corresponding virtual finger according to the real 3D coordinates. Optionally, after the user lifts their finger, it may first leave the touch area surface vertically, in which case the virtual finger display position will not change. Based on this, after detecting the lift-off event, the remote control may initially maintain the virtual finger's locked state, and then unlock the virtual finger if the remote control determines that the user's finger height is greater than or equal to a certain height threshold. This certain height threshold may, for example, be less than or equal to the height threshold 1 used to determine whether to lock the virtual finger during the user's finger hovering and descending, such as half of the height threshold 1. In this way, during the virtual finger locking process, the remote control device can also be unlocked via virtual finger, responding to user operations in a timely manner and improving the user experience.

[0166] For example, after the virtual finger is locked but before the user's finger touches the touch area of ​​the remote control device, if the remote control device detects an increase in the height of the user's finger—for instance, the user stops pressing down on their finger and begins to lift it—then the remote control device can unlock the virtual finger if it determines that the user's finger height has increased to a level greater than or equal to a height threshold 1. This satisfies the user's need to perform other touch operations and, through the limitation of height threshold 1, avoids the problem of misunderstanding the unlocking of the virtual finger due to slight up-and-down movements during the user's finger pressing process. Optionally, the height threshold 1 for unlocking the virtual finger by lifting the user's finger can be the same as or different from the height threshold 1 for determining whether the virtual finger is locked during the user's finger's hovering and descent.

[0167] The above sections introduced the virtual finger locking and unlocking processes. The following sections, using flowcharts, provide a detailed explanation of the overall process of virtual finger locking and unlocking.

[0168] Figure 20 is a schematic flowchart of a device control method provided in an embodiment of this application. It should be noted that this method is not limited to the specific order shown in Figure 20 and below. It should be understood that in other embodiments, the order of some steps in this method can be interchanged according to actual needs, or some steps can be omitted or deleted. The method includes the following steps:

[0169] S1001, The remote control device detected the position of the user's finger.

[0170] In some embodiments, the space within a preset height range above the touch area of ​​the remote control device and the touch area of ​​the remote control device constitute the detection area (i.e., detection range) of the remote control device, as shown in Figure 7. After the user's finger enters the detection area, the remote control device can obtain the position of the user's finger based on the changes in the acquired detection data. Optionally, the finger position may be, for example, the three-dimensional coordinates of the center of the finger. Optionally, the detection data may be, for example, capacitance data. It should be understood that the remote control device can also obtain the position of the user's finger through various methods such as ultrasonic detection, infrared detection, and camera capture, which will not be exemplified in this embodiment.

[0171] Furthermore, the remote control device can predict the position of the virtual finger based on the scheme described above. After the prediction is completed, the remote control device can continue to execute step S1002.

[0172] S1002. The remote control device determines whether the locking conditions are met. If the locking conditions are met, proceed to step S1003; if the locking conditions are not met, proceed to step S1001.

[0173] S1003. The remote control device sets the virtual finger to lock at the location where the user intends to interact.

[0174] In some embodiments, after the remote control device predicts the position of the virtual finger, it can continue to acquire the position of the user's finger above the remote control device. After acquiring the position of the user's finger, the remote control device can determine whether the current frame meets the locking condition based on the position of the user's finger. If the locking condition is met, the remote control device can determine that the three-dimensional coordinates indicated by the current frame correspond to the position where the user intends to perform a touch operation, and can lock the virtual finger corresponding to the three-dimensional coordinates (i.e., the predicted position of the virtual finger), such as setting the locking state of the virtual finger (as in step S1003). If the locking condition is not met, the remote control device can choose not to lock the virtual finger corresponding to the acquired three-dimensional coordinates, but instead report the real three-dimensional coordinates (or display position, i.e., the position of the virtual finger mapped based on the position of the user's finger projected above the remote control device) to the display device in real time, as in step S1001.

[0175] Optionally, the locking conditions may include, for example, the user's finger height being less than a height threshold 1, and the user's finger moving downward in the z-axis direction being greater than a z-axis direction moving speed threshold, moving in the x-axis direction being less than an x-axis direction moving speed threshold, and moving in the y-axis direction being less than a y-axis direction moving speed threshold.

[0176] Optionally, the locked state of the virtual finger may include fixing the display position of the virtual finger. Optionally, by fixing the x-axis and y-axis coordinates in a three-dimensional coordinate system, the same x-axis and y-axis coordinates can be used to obtain the same display position. Combined with the same movement distance and direction, the virtual finger is displayed at the same position, thus achieving the locked state of the virtual finger. For example, after determining the locked state of the virtual finger corresponding to the user's intended interaction position, the remote control device can subsequently report the same three-dimensional coordinates to the display device, or report the same x-axis coordinates, the same y-axis coordinates, and the actual z-axis coordinates. This allows the display device to obtain the same virtual finger display position based on the obtained three-dimensional coordinates combined with the movement distance and direction, until the unlocking conditions are subsequently met. For example, after the remote control device determines the locked state of the virtual finger corresponding to the user's intended interaction position, it can use the same three-dimensional coordinates, or the same x-axis coordinates, the same y-axis coordinates, and the real z-axis coordinates, to obtain the same virtual finger display position. After the virtual finger display position is reported to the display device, the display device is triggered to display the virtual finger at the same virtual finger display position until the unlocking conditions are subsequently confirmed.

[0177] For example, the remote control device is configured with a height threshold of z. thresh The threshold velocity vz in the z-axis direction is 5 mm. threshThe threshold velocity vx in the x-axis direction is -50 mm / s. thresh The y-axis movement speed threshold is 30 mm / s. As shown in Table 1, the remote control device obtains the three-dimensional coordinates of the user's finger in frame m-4 as (97.4, 26.5, 8.3). At this time, the user's finger height is 8.3 mm, which is greater than the height threshold of 1. The remote control device can determine that the locking condition is not met and can trigger the acquisition of the virtual finger display position based on the actual user finger position. However, based on the changes in the data of the next few consecutive frames, the remote control device can determine that the user's finger height begins to decrease. Then, in frame m, the remote control device obtains the three-dimensional coordinates of the user's finger as (97.5, 25.9, 4.9), and the corresponding three-dimensional velocity is (1.351, -0.564, -104.47). Subsequently, the remote control device, based on the aforementioned formula group one, determines that the finger height is 4.9mm < 5mm, the downward movement speed along the z-axis is -104.47mm / s < -50mm / s, the movement speed along the x-axis is 1.351mm / s < 30mm / s, and the movement speed along the y-axis is 0.564mm / s < 30mm / s, thus confirming that the locking conditions are met. Therefore, as shown in Table 1, starting from the m-th frame, even if the virtual finger projection position changes, the actual display position of the virtual finger obtained by the remote control device or display device will not change. The virtual finger projection position is obtained by transforming the three-dimensional coordinates of the real user's finger position, and the actual display position of the virtual finger is obtained by combining the three-dimensional coordinates of the m-th frame that meets the locking conditions with the movement distance and direction. Optionally, the remote control device reports the locked three-dimensional coordinates to the display device, or the remote control device reports the display position obtained after transforming the locked three-dimensional coordinates, thereby enabling the display device to display the virtual finger at the same display position, thus achieving virtual finger locking.

[0178] Table 1

[0179] For example, the remote control device is configured with a height threshold of z. thresh The threshold velocity vz in the z-axis direction is 5mm. thresh The threshold velocity vx in the x-axis direction is -50 mm / s. threshThe y-axis movement speed threshold is 30 mm / s. As shown in Table 2, the remote control device obtains the three-dimensional coordinates of the user's finger in frame m-4 as (76.5, 13.5, 9.9). At this time, the user's finger height is 9.9 mm, which is greater than the height threshold of 1. The remote control device can determine that the locking condition is not met and can trigger the acquisition of the virtual finger display position based on the actual user finger position. However, based on the changes in data of several subsequent frames, the remote control device can determine that the user's finger height begins to decrease. Then, in frame m, the remote control device obtains the three-dimensional coordinates of the user's finger as (76, 13, 4.8), and the corresponding three-dimensional velocity is (-2.633, -1.251, -195.61). Subsequently, the remote control device, based on the aforementioned formula group one, determines that the finger height is 4.8mm < 5mm, the downward movement speed along the z-axis is -195.61mm / s < -50mm / s, the movement speed along the x-axis is 2.633mm / s < 30mm / s, and the movement speed along the y-axis is 1.251mm / s < 30mm / s, thus confirming that the current locking condition is met. Therefore, as shown in Table 2, starting from the m-th frame, even if the virtual finger projection position changes, the actual display position of the virtual finger obtained by the remote control device or display device will not change. The virtual finger projection position is obtained by transforming the three-dimensional coordinates of the actual user's finger position, and the actual display position of the virtual finger is obtained by combining the three-dimensional coordinates of the m-th frame that meets the locking condition with the movement distance and direction. Optionally, the remote control device reports the locked three-dimensional coordinates to the display device, or the remote control device reports the display position obtained after transforming the locked three-dimensional coordinates to the display device, thereby enabling the display device to display the virtual finger at the same display position, thus achieving virtual finger locking.

[0180] Table 2

[0181] S1004. The remote control device determines whether the unlocking conditions are met. If the unlocking conditions are met, proceed to step S1005; if the unlocking conditions are not met, proceed to step S1003.

[0182] S1005. The remote control device is set to the unlocked state of the virtual finger corresponding to the user's intended interaction position.

[0183] In some embodiments, while the virtual finger is in a locked state, the remote control device can also determine whether the unlocking conditions are met and whether the virtual finger needs to be unlocked based on changes in the movement state of the user's finger, such as setting the virtual finger to an unlocked state. Optionally, the remote control device can determine changes in the movement state of the user's finger based on changes in the three-dimensional coordinates of the user's finger.

[0184] Optionally, the unlocking conditions may include one or more of the following: no touch event detected within a preset time, x-axis offset exceeding a horizontal offset threshold, y-axis offset exceeding a vertical offset threshold, or detection of a lift event.

[0185] For example, if the remote control device detects that the unlocking condition is met, it can trigger virtual finger unlocking, as in step S1005. For example, the remote control device performs coordinate transformation based on the actual three-dimensional coordinates of the user's finger to obtain the virtual finger display position. For example, in response to meeting the unlocking condition, the remote control device sends the actual three-dimensional coordinates obtained in the current frame to the display device, causing the display device to perform coordinate transformation based on the actual three-dimensional coordinates to obtain the virtual finger display position, thereby ensuring the virtual finger display position meets the user's needs. As another example, in response to meeting the unlocking condition, the remote control device can perform coordinate transformation based on the actual three-dimensional coordinates obtained in the current frame to obtain the virtual finger display position, and send this virtual finger display position to the display device to trigger virtual finger display, thereby ensuring the virtual finger display position meets the user's needs. In the unlocked state after the virtual finger is unlocked, the projected position of the virtual finger is the same as the actual display position of the virtual finger.

[0186] For example, if the remote control device detects that the current unlocking conditions are not met, it can continue to keep the virtual finger locked, such as by executing step S1003.

[0187] In this way, the remote control device can unlock the virtual finger in a timely manner according to the unlocking conditions. By flexibly locking and unlocking the virtual finger, it can meet the user's needs and improve the user experience.

[0188] In some embodiments, the locking state of the virtual finger may include, for example, a hover lock state and a touch lock state. The hover lock state means that the virtual finger corresponding to the user's finger position is locked while the user's finger is hovering. The touch lock state means that the virtual finger corresponding to the user's finger position is locked while the user's finger touches the touch area of ​​the remote control device.

[0189] Optionally, different unlocking conditions can be configured corresponding to different virtual finger lock states. For example, as shown in Figure 21, the virtual finger locking and unlocking process of steps S1003 and S1004 described above will be explained in detail below through steps S1101-S1104.

[0190] S1101, The remote control device sets the hover lock state of the virtual finger corresponding to the user's intended interaction position.

[0191] In some embodiments, while the user's finger is moving within the detection range of the remote control device, the remote control device determines that the locking conditions are met, as described in step S1002 above. Then, the remote control device can set the hovering lock state of the virtual finger corresponding to the user's intended interaction position, thereby locking the virtual finger.

[0192] For example, as shown in Table 1 above, in the m-th frame, when the user's finger height is 4.9mm, the remote control device determines that the locking condition is met and sets the virtual finger to a floating lock state. Subsequently, during the user's finger's floating descent action, the actual display position of the virtual finger no longer changes.

[0193] S1102. The remote control device detects a touch event within a preset time and the user's intended interaction location is within a certain range. If yes, proceed to step S1103; otherwise, proceed to step S1005.

[0194] S1103. The remote control device sets the touch lock state of the virtual finger corresponding to the user's intended interaction position.

[0195] In some embodiments, the remote control device predicts that a user's finger may touch the screen within a preset time period. Therefore, it locks the virtual finger in advance to prevent errors in user operation from causing the virtual finger's display position to deviate from the user's intended position for clicking or other operations. If the remote control device detects the touch event within the preset time period, it can determine that the prediction of the user's intention is correct and can maintain the virtual finger's lock. Furthermore, after detecting the touch event, it can set the virtual finger to a touch-locked state (as in step S1103). Otherwise, if the touch device fails to detect the touch event within the preset time period, it can determine that the prediction of the user's intention is incorrect. In this case, the remote control device needs to unlock the virtual finger, such as setting it to an unlocked state (as in step S1005).

[0196] Optionally, while the virtual finger is in a hover-locked state, if the user's finger moves beyond a certain range around the locked 3D coordinates in the x-axis and / or y-axis planes, such as the x-axis offset exceeding a lateral offset threshold and / or the y-axis offset exceeding a longitudinal offset threshold, then the remote control device can determine that the prediction of the user's intention is incorrect. In this case, the remote control device needs to unlock the virtual finger, such as setting the virtual finger to an unlocked state (e.g., executing step S1005). For example, if the user's finger moves to the left beyond a preset range near the x-axis coordinates in the 3D coordinates, the user's intention may be to move the virtual finger to the left, rather than to perform a click operation. In this situation, the remote control device needs to promptly unlock the virtual finger to meet the user's virtual finger movement needs.

[0197] For example, in the scenario shown in Table 1 above, the remote control device determines that the locking condition is met in the m-th frame, and the height z of the z-axis in the three-dimensional coordinates obtained in the m-th frame is... m The distance is 4.9mm, and the speed vz of the user's finger along the z-axis is obtained in the m-th frame. m The frame rate is -104.47 mm / s, and the refresh rate fs of the remote control device is 120 Hz. Therefore, the remote control device obtains the preset number of frames according to Formula 1 above. In other words, if the remote control device determines that a touch event needs to be detected within the next 6 frames (e.g., no later than frame m+6), it does not need to unlock the virtual finger; otherwise, it needs to set the virtual finger to an unlocked state. For example, as shown in Table 1 above, if the 3D coordinates obtained by the remote control device in frame m+2 are (97.9, 25.7, 0), and this 3D coordinate indicates that the user's finger height is 0, then the remote control device determines that a touch event has been detected. If the remote control device determines that a touch event has been detected within frame k, that is, within a preset time, then, as shown in Table 1 above, the remote control device can continue to maintain the virtual finger's locked state, such as switching the virtual finger to touch-locked state, and ensuring that the actual display position of the virtual finger does not change with the change of the virtual finger's projected position.

[0198] Thus, by locking the virtual finger position in a timely manner, even if the user's finger cannot strictly execute a vertical descent, the virtual finger click position will not deviate. For example, as shown in Table 1 above, starting from frame m, the user's finger position continuously descends. In frame m+2, the user's finger touches the touch area of ​​the remote control device. At this time, the virtual finger projection position has deviated from the virtual finger display position that the user intended to trigger the virtual finger click in frame m. However, due to the locking of the virtual finger, the display device displays the virtual finger according to the locked virtual finger display position. Therefore, in frame m+2, the actual display position of the virtual finger is the same as the virtual finger display position that the user intended to trigger the virtual finger click in frame m, satisfying the user's click requirement.

[0199] For example, as shown in Table 2 above, the remote control device determines that the locking condition is met in the m-th frame, and the height z of the z-axis in the three-dimensional coordinates obtained in the m-th frame is... m The distance is 4.8mm, and the speed vz of the user's finger along the z-axis is obtained in the m-th frame. m The frame rate is -195.61 mm / s, and the refresh rate (fs) of the remote control device is 120 Hz. Therefore, the remote control device obtains the preset frame rate according to Formula 1 above. In other words, if the remote control device determines that a touch event needs to be detected within the next 3 frames (e.g., no later than frame m+3), it does not need to unlock the virtual finger; otherwise, it needs to set the virtual finger to an unlocked state. For example, as shown in Table 2 above, the 3D coordinates obtained by the remote control device in frame m+3 are (74.5, 12, 5.6), which indicates that the user's finger height is 5.6mm. Therefore, the remote control device can determine that no touch event was detected within the preset time. Thus, as shown in Table 2 above, the remote control device can unlock the virtual finger in frame m+3, setting the virtual finger to an unlocked state. Furthermore, starting from frame m+3, the actual display position of the virtual finger is the same as its projected position. That is, the remote control device or display device performs coordinate transformation based on the actual 3D coordinates of the user's finger position to obtain the display position of the virtual finger, enabling the virtual finger display position to change with the user's finger position.

[0200] Thus, if the user's intention is misjudged, the remote control device can promptly unlock the virtual finger. The device control method provided in this application embodiment can unlock the virtual finger more quickly, improving the user experience.

[0201] S1104. The remote control device determines that the user's intended interaction position is within a certain range and no lift event is detected. If yes, proceed to step S1103; otherwise, proceed to step S1005.

[0202] In some embodiments, while the virtual finger is in a touch-locked state, if the user lifts their hand, causing their finger to leave the touch area of ​​the remote control device, it can be determined that the user-instructed click event has been completed; or, if the user's finger moves a distance exceeding a certain range, it can be determined that the user has instructed to perform an operation such as a swipe. Then, in response to detecting a lift-off event, or if the user's finger moves beyond a certain range around the locked three-dimensional coordinates in the x-axis and / or y-axis planes (e.g., the x-axis offset exceeds a lateral offset threshold and / or the y-axis offset exceeds a longitudinal offset threshold), the remote control device can release the touch-locked state of the virtual finger, such as setting the virtual finger to an unlocked state (e.g., executing step S1005).

[0203] Alternatively, while the virtual finger is in a touch-locked state, if the user's finger moves within a certain range—for example, the movement of the user's finger in the x-axis and y-axis planes does not exceed a certain range around the locked three-dimensional coordinates, and it is not lifted—then the remote control device can determine that the user's intended interaction position is within a certain range and no lift-off event is detected. Therefore, the remote control device can maintain the touch-locked state of the virtual finger (as in step S1103). For example, if the user intends to perform a long press operation, in response to this operation, the remote control device can maintain the lock on the virtual finger to meet the user's usage needs.

[0204] Optionally, the lift event may include detecting a user's finger lift operation; or, the lift event may include detecting that after the user's finger lift operation, the finger height exceeds a certain height threshold.

[0205] In this way, the remote control device can flexibly set the virtual finger to lock or unlock according to the user's operation, meet the user's needs, and improve the user experience.

[0206] In some embodiments, the above description uses the interaction process between a remote control device and a display device to illustrate the device control method provided in this application. In some examples, the remote control device and the display device may be the same device; for example, the touch area of ​​the remote control device may have a display function. For instance, the remote control device may be an electronic device with a touchscreen, such as a mobile phone or tablet. In this case, the methods described in the above embodiments are also applicable to the touchscreen electronic device.

[0207] For example, an electronic device detects that a user's finger has entered the detection range. As the user's finger descends, the electronic device can lock the display position of the virtual finger being displayed, thereby preventing accidental touches.

[0208] Optionally, the specific implementation methods of the virtual finger locking process and the virtual finger unlocking process of the touch screen electronic device can refer to the methods described in the above embodiments, and will not be repeated here.

[0209] For example, Figure 22 shows a schematic flowchart of a device control method provided in an embodiment of this application, the method including the following steps:

[0210] S2201. When the height of the user's finger from the remote control device is a first height, obtain the first position information corresponding to the projection position of the user's finger on the remote control device.

[0211] It can be understood that the projection position of the user's finger on the remote control device is the same as the projection position of the user's finger above the remote control device mentioned above. For example, this projection position can be represented by two-dimensional coordinates (x, y) in the coordinate system of the remote control device shown in Figure 11.

[0212] Optionally, the first position information may be coordinates on the display device. Alternatively, it may be information used to determine the coordinates on the display device, such as the projection position of the user's finger on the remote control device, or at least one of the size ratios between the remote control device and the display device.

[0213] The remote control device establishes a communication connection with the display device, and the remote control device is used to control the display device.

[0214] S2202. Determine the second position on the display device based on the first position information and the preset movement information corresponding to the first position information.

[0215] The second position is used to determine the object to be manipulated on the display device. For example, the second position can be represented by two-dimensional coordinates (x, y) in the display device coordinate system as shown in Figure 11. Optionally, the second position may not be the position of the object to be manipulated. For example, when the object to be manipulated is a control, a display screen, or other object occupying a large display area, the position of the object to be manipulated can be the position of a certain display area, and the object to be manipulated can be determined based on the display area where the second position is located. Alternatively, the second position can also be the position of the object to be manipulated. For example, when the object to be manipulated is handwriting, a cursor, a blank point, or other object whose position can be directly represented by a point coordinate, the second position can be the position of the object to be manipulated.

[0216] In some embodiments, steps S2201 to S2202 can be executed by either a remote control device or a display device. For example, when steps S2201 to S2202 are all executed by a remote control device, the first position information can be coordinates on the display device. The remote control device can directly send the first position information, the second position, etc., to the display device so that the display device can present the virtual finger as described above. As another example, when steps S2201 to S2202 are all executed by the display device, the first position information can also be coordinates on the display device. The remote control device can send the projection position of the user's finger on the remote control device to the display device so that the display device can determine the first position information based on the projection position. Optionally, in this example, the preset movement information corresponding to the first position information can be determined by the display device or sent to the display device by the remote control device.

[0217] In other embodiments, any one of steps S2201 to S2202 described above can be executed by a remote control device, and the other step can be executed by a display device. For example, taking the case where step S2201 is executed by a remote control device and step S2202 is executed by a display device, after obtaining the first location information, the remote control device can send the first location information to the display device, and the display device can determine the second location based on the received first location information. Similarly, in this example, the preset movement information corresponding to the first location information can also be determined by the display device, or it can be sent to the display device by the remote control device.

[0218] For example, in conjunction with the example shown in Figure 12, the first position information can be position B2 as shown in Figure 12 (4), and the second position can be position C2 as shown in Figure 12 (6).

[0219] In some embodiments, step S2201 can be specifically implemented as follows: when the first height is less than or equal to a first height threshold, acquiring first position information corresponding to the projection position of the user's finger on the remote control device when the height of the user's finger from the remote control device is the first height threshold. That is, when the first height is less than the first height threshold, the first position information is also determined based on the projection position of the user's finger on the remote control device when the height of the user's finger from the remote control device is the first height threshold. In other words, whether the height of the user's finger from the remote control device is less than or equal to the first height threshold, the determined projection position of the user's finger on the remote control device is the same. For example, in the embodiment shown in FIG22, the first height threshold can be the height threshold 1 described above.

[0220] In some embodiments, when the height of the user's finger from the remote control device is greater than or equal to a first height threshold, a third position corresponding to the projection position of the user's finger on the remote control device is determined as the position of the object to be manipulated on the display device. For example, similar to the second position, the third position can also be represented by two-dimensional coordinates (x, y) in the display device coordinate system as shown in Figure 11, etc. Referring to the example shown in Figure 12, the third position can be position A2 as shown in (2) of Figure 12. For example, the position of the object to be manipulated can be the display position of the virtual finger. Similarly, this step can also be performed by the remote control device and / or the display device.

[0221] In some embodiments, the preset movement information may include at least one of a preset movement distance and a preset movement direction. The preset movement distance may be the movement distance described above, and the preset movement direction may be the movement direction described above. For example, the preset movement direction may be represented by a direction angle, such as 20 degrees east of south, 50 degrees west of north, etc.

[0222] In some embodiments, when the user's finger is at a second height from the remote control device, second position information corresponding to the projection position of the user's finger on the remote control device is obtained. Optionally, the first height and the second height can be the same or different. The second height can also be less than or equal to a first height threshold. When the user's finger is at the second height from the remote control device, the projection position of the user's finger on the remote control device can be different from the projection position of the user's finger on the remote control device when the user's finger is at the first height.

[0223] In some implementations, the first position represented by the first position information and the second position represented by the second position information are located in different display areas on the display device. For example, the first position is located in the upper left area of ​​the display device, and the second position is located in the lower right area of ​​the display device.

[0224] In some implementations, the user operation corresponding to the first location information differs from the user operation corresponding to the second location information. This is understandable. The user operation corresponding to the first location information can refer to the operation used to manipulate the object to be manipulated, determined based on the first location information and the preset movement information corresponding to the first location information. Similarly, the user operation corresponding to the second location information can refer to the operation used to manipulate the object to be manipulated, determined based on the second location information and the preset movement information corresponding to the second location information. For example, the user operation corresponding to the first location information could be a click, and the user operation corresponding to the second location information could be a long press. Another example is that the user operation corresponding to the first location information could be a double-click, and the user operation corresponding to the second location information could be dragging, etc.

[0225] In some implementations, the finger used by the user to obtain the first location information is different from the finger used to obtain the second location information. For example, the user may use their thumb to obtain the first location information and their index finger to obtain the second location information.

[0226] In the three implementations described above, the preset movement information corresponding to the first position information is different from the preset movement information corresponding to the second position information. Optionally, at least one of the preset movement distance and preset movement direction may be different.

[0227] For example, when the preset movement distance and preset movement direction are determined by the AI ​​model, the AI ​​model can also combine one or more of the following: the area on the display device represented by the acquired location information (such as the first location information and the second location information mentioned above), the user's operation type (such as clicking, swiping, dragging, etc.), and the finger used by the user when performing the operation, to determine at least one of the movement distance and movement direction corresponding to the acquired location information. Similarly, when the preset movement distance and preset movement direction are determined by a stored mapping relationship, the mapping relationship can also include one or more of the following: the area on the display device represented by the acquired location information (such as the first location information and the second location information mentioned above), the user's operation type (such as clicking, swiping, dragging, etc.), and the finger used by the user when performing the operation, and the mapping relationship between the acquired location information and the movement distance and movement direction corresponding to that location information.

[0228] For other implementations regarding the second height of the user's finger from the remote control device, please refer to the implementation when the user's finger is at the first height of the remote control device.

[0229] In some embodiments, after executing step S2202, the operation can also be performed on the object to be manipulated, determined based on the second position, when the height of the user's finger from the remote control device is 0. Alternatively, the operation can also be performed on the object to be manipulated, determined based on the second position, when the height of the user's finger from the remote control device is less than or equal to a second height threshold. In some implementations, the second height threshold can be greater than 0 and less than or equal to a first height threshold. In other implementations, the second height threshold and the first height threshold can be the same. In this implementation, when the height of the user's finger from the remote control device is less than or equal to the first height threshold, not only is the object to be manipulated on the display device predicted, but the object to be manipulated is also manipulated. For example, the second height threshold can be height threshold 2 as described above. Similarly, the operation in this embodiment can also be performed by the remote control device and / or the display device. Furthermore, the display device can also perform operations based on the object to be manipulated determined by the second position, such as responding to manipulation of the object to be manipulated.

[0230] In some implementations, the object to be manipulated, determined based on the second location, can be a first control. For example, referring to the example shown in Figure 12, the first control can be control C2. Even more exemplarily, the first control can be any control as shown in Figure 8. Manipulating the first control can include clicking, long-pressing, etc.

[0231] In other implementations, the display device may have multiple displays, with the object to be manipulated determined based on the second location being the first display among the multiple displays. For example, referring to the example shown in Figure 17, the first display may be display 2. Referring to the example shown in Figure 18, the first display may be display 4. Manipulating the first display may involve methods such as displaying the image on the first display, highlighting the first display, etc.

[0232] In some embodiments, as the user's finger descends from a first height to a third height from the height of the remote control device, the position of the object to be manipulated on the display device is determined to be a second position. That is, as the user's finger descends from the first height to the third height from the height of the remote control device, the second position is locked, which is also the projection position of the user's finger on the remote control device when the user's finger is at the first height, i.e., the object to be manipulated is locked. The third height is the height at which manipulation of the object determined based on the second position is triggered. For example, the third height can be 0, or it can be less than or equal to the aforementioned second height threshold. Similarly, this step can also be performed by the remote control device and / or the display device.

[0233] In some embodiments, before determining that the position of the object to be manipulated on the display device is always the second position, it may be determined whether the locking condition described above is met. If the locking condition is met, then the position of the object to be manipulated on the display device is determined to be always the second position. If the locking condition is not met, the position of the object to be manipulated on the display device may be determined based on the actual position of the user's finger projected on the remote control device and the size ratio between the display device and the remote control device.

[0234] The above primarily describes the solutions provided by the embodiments of this application from a methodological perspective. It is understood that, in order to achieve the above functions, the remote control device and / or display device includes corresponding hardware structures and / or software modules for executing each function. By combining the units and algorithm steps of the various examples described in the embodiments disclosed in this application, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by a computer driving the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solutions of the embodiments of this application.

[0235] This application provides embodiments that can divide remote control devices and / or display devices into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0236] Figure 23 shows a schematic diagram of a remote control device provided in an embodiment of this application. The remote control device 2300 can be used to implement the methods executed by the remote control device described in the above method embodiments. For example, the remote control device 2300 may include a processing unit 2301 and a communication unit 2302.

[0237] The processing unit 2301 is used to support the remote control device 2300 in performing the processing function of the remote control device as described in any one of Figures 1 to 22, and the communication unit 2302 is used to support the remote control device 2300 in performing the communication function of the remote control device as described in any one of Figures 1 to 22.

[0238] Optionally, the remote control device 2300 shown in FIG23 may further include a storage unit 2303, which stores programs or instructions. When the processing unit 2301 executes the program or instructions, the remote control device 2300 shown in FIG23 can perform the method described in the above-described method embodiments.

[0239] Figure 24 shows a schematic diagram of a display device provided in an embodiment of this application. The display device 2400 can be used to implement the methods executed by the display device described in the above method embodiments. For example, the display device 2400 may include a processing unit 2401, a communication unit 2402, and a display unit 2403.

[0240] The processing unit 2401 is used to support the display device 2400 in performing the processing function of the display device as described in any one of Figures 1 to 22; the communication unit 2402 is used to support the display device 2400 in performing the communication function of the display device as described in any one of Figures 1 to 22; and the display unit 2403 is used to support the display device 2400 in performing the display function of the display device as described in any one of Figures 1 to 22.

[0241] Optionally, the display device 2400 shown in FIG. 24 may further include a storage unit (not shown in FIG. 24) storing a program or instructions. When the processing unit 2401 executes the program or instructions, the display device 2400 shown in FIG. 24 can perform the method described in the above-described method embodiments.

[0242] The technical effects of the remote control device 2300 shown in Figure 23 or the display device 2400 shown in Figure 24 can be referred to the technical effects described in the above method embodiments, and will not be repeated here. The processing unit involved in Figure 23 or Figure 24 can be implemented by a processor or processor-related circuit components, and can be a processor or processing module. The communication unit can be implemented by a transceiver or transceiver-related circuit components, and can be a transceiver or transceiver module. The display unit can be implemented by display screen-related components.

[0243] This application also provides a chip system, as shown in FIG25, which includes at least one processor 2501 and at least one interface circuit 2502. The processor 2501 and the interface circuit 2502 are interconnected via lines. For example, the interface circuit 2502 can be used to receive signals from other devices. As another example, the interface circuit 2502 can be used to send signals to other devices (e.g., the processor 2501). Exemplarily, the interface circuit 2502 can read instructions stored in a memory and send those instructions to the processor 2501. When the instructions are executed by the processor 2501, the remote control device can perform the various steps performed by the remote control device in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application.

[0244] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0245] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0246] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0247] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0248] This application also provides a computer storage medium storing computer instructions that, when executed on a remote control device and / or a display device, cause the remote control device and / or the display device to perform the methods described in the above-described method embodiments.

[0249] This application provides a computer program product, which includes a computer program or instructions that, when run on a computer, cause the computer to perform the methods described in the above-described method embodiments.

[0250] In addition, this application also provides an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the apparatus to perform the methods in the above-described method embodiments.

[0251] In this embodiment, the remote control device and / or display device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0252] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0253] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The embodiments can be combined with or referenced to each other without conflict. The apparatus embodiments described above are merely illustrative; for example, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0254] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0256] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0257] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A device manipulation method, characterized by, The method includes: When the user's finger is at a first height from the remote control device, first position information corresponding to the projection position of the user's finger on the remote control device is acquired. A second position on the display device is determined based on the first position information and the preset movement information corresponding to the first position information, and the second position is used to determine the object to be manipulated on the display device; The remote control device establishes a communication connection with the display device, and the remote control device is used to control the display device.

2. The method of claim 1, wherein, When the user's finger is at a first height from the remote control device, first position information corresponding to the projection position of the user's finger on the remote control device is acquired, including: When the first height is less than or equal to the first height threshold, first position information corresponding to the projection position of the user's finger on the remote control device is obtained when the height of the user's finger from the remote control device is the first height threshold.

3. The method of claim 2, wherein, The method further includes: When the height of the user's finger from the remote control device is greater than or equal to the first height threshold, the third position corresponding to the projection position of the user's finger on the remote control device is determined as the position of the object to be manipulated on the display device.

4. The method according to any one of claims 1-3, characterized in that, The preset movement information includes at least one of preset movement distance and preset movement direction.

5. The method of claim 4, wherein, The method further includes: When the user's finger is at a second height from the remote control device, second position information corresponding to the projection position of the user's finger on the remote control device is acquired; the first position represented by the first position information and the second position represented by the second position information are in different display areas on the display device, or the user operation corresponding to the first position information is different from the user operation corresponding to the second position information, or the finger used by the user when acquiring the first position information is different from the finger used by the user when acquiring the second position information; the preset movement information corresponding to the first position information is different from the preset movement information corresponding to the second position information.

6. The method according to any one of claims 1-5, characterized in that, After determining the second position on the display device based on the first position information and the preset movement information corresponding to the first position information, the method further includes: When the user's finger is 0 meters away from the remote control device, the object to be controlled is manipulated based on the second position.

7. The method according to any one of claims 1-5, characterized in that, After determining the second position on the display device based on the first position information and the preset movement information corresponding to the first position information, the method further includes: When the height of the user's finger from the remote control device is less than or equal to a second height threshold, the object to be manipulated is manipulated based on the second position, wherein the second height threshold is greater than 0 and less than or equal to a first height threshold.

8. The method according to any one of claims 1-7, characterized in that, The object to be manipulated, determined based on the second position, is the first control.

9. The method according to any one of claims 1-7, characterized in that, The display device has multiple displays, and the object to be manipulated determined based on the second position is the first display among the multiple displays.

10. The method according to any one of claims 1-9, characterized in that, The method is performed by the remote control device or the display device.

11. The method according to any one of claims 1-9, characterized in that, The first location information is obtained by the remote control device, and the second location is determined by the display device.

12. The method according to any one of claims 1-11, characterized in that, The method further includes: As the user's finger descends from the first height to the third height from the height of the remote control device, the position of the object to be manipulated on the display device is determined to be the second position, wherein the third height is the height at which manipulation of the object to be manipulated is triggered based on the second position.

13. A system, characterized by The system includes a remote control device and a display device, the remote control device and the display device establishing a communication connection, and the remote control device being used to control the display device; The remote control device is used to acquire first position information corresponding to the projection position of the user's finger on the remote control device when the user's finger is at a first height from the remote control device; The remote control device or the display device is used to determine a second position on the display device based on the first position information and the preset movement information corresponding to the first position information, wherein the second position is used to determine the object to be controlled on the display device; The display device is used to perform an operation on the object to be manipulated as determined by the second position.

14. The system of claim 13, wherein, The remote control device is specifically used to acquire, when the first height is less than or equal to the first height threshold, first position information corresponding to the projection position of the user's finger on the remote control device when the height of the user's finger from the remote control device is the first height threshold.

15. The system of claim 14, wherein, The remote control device or the display device is further configured to determine the third position corresponding to the projection position of the user's finger on the remote control device as the position of the object to be manipulated on the display device when the height of the user's finger from the remote control device is greater than or equal to the first height threshold.

16. The system of any of claims 13-15, wherein, The preset movement information includes at least one of preset movement distance and preset movement direction.

17. The system according to claim 16, characterized in that, The remote control device is further configured to acquire second position information corresponding to the projection position of the user's finger on the remote control device when the user's finger is at a second height from the remote control device; the first position represented by the first position information and the second position represented by the second position information are located in different display areas on the display device, or the user operation corresponding to the first position information is different from the user operation corresponding to the second position information, or the finger used by the user when acquiring the first position information is different from the finger used by the user when acquiring the second position information; the preset movement information corresponding to the first position information is different from the preset movement information corresponding to the second position information.

18. The system of any of claims 13-17, wherein, The remote control device is also used to control the object to be controlled based on the second position when the height of the user's finger from the remote control device is 0.

19. The system of any of claims 13-17, wherein, The remote control device is further configured to control the object to be controlled based on the second position when the height of the user's finger from the remote control device is less than or equal to a second height threshold, wherein the second height threshold is greater than 0 and less than or equal to a first height threshold.

20. The system of any of claims 13-19, wherein, The object to be manipulated, determined based on the second position, is the first control.

21. The system of any of claims 13-19, wherein, The display device has multiple displays, and the object to be manipulated determined based on the second position is the first display among the multiple displays.

22. The system of any of claims 13-21, wherein, The remote control device is further configured to determine that the position of the object to be manipulated on the display device is the second position as the distance between the user's finger and the remote control device decreases from the first height to the third height, wherein the third height is the height at which manipulation of the object to be manipulated is triggered based on the second position.

23. A remote control device, characterized by include: The remote control device comprises a processor, a communication interface, and a memory, wherein the memory and the communication interface are coupled to the processor, the communication interface is used to communicate with other devices, and the memory is used to store computer program code, the computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the remote control device performs the method as described in any one of claims 1-12.

24. A display device, characterized by include: The device includes a processor, a communication interface, a display screen, and a memory, wherein the memory, the communication interface, and the display screen are coupled to the processor, the communication interface is used to communicate with other devices, and the memory is used to store computer program code, the computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the display device performs the method as described in any one of claims 1-12.

25. A computer readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on a remote control device or display device, causes the remote control device or display device to perform the method as described in any one of claims 1-12.

26. A computer program product, characterised in that, The computer program product includes: a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-12.

Images