Human-machine interaction method, electronic device, and system

Abstract

Embodiments of the present application provide a human-machine interaction method, an electronic device, and a system. The method comprises: a first device displaying a first interface of a first application, wherein the first application is a third-party application; on the basis of the system capability of the first device, acquiring position information of one or more controls present on the first interface, wherein the one or more controls include a first control; and when it is determined, on the basis of the position information of the first control, that a user operates the first control, the first device providing first feedback. By means of the method, the electronic device, and the system, upon detecting that a user clicks a control on an interface of a current third-party application, an electronic device can provide corresponding feedback on the basis of the system capability of the electronic device, so that interaction feedback between the user and the electronic device is independent of the third-party application, thereby delivering timely and rich feedback to the user and providing an immersive interaction experience for the user.

Description

Human-computer interaction methods, electronic devices and systems

[0001] This application claims priority to Chinese Patent Application No. 202510198822.3, filed on February 21, 2025, entitled "Method, Electronic Device and System for Human-Computer Interaction", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of human-computer interaction, and more specifically, to a method, electronic device, and system for human-computer interaction. Background Technology

[0003] As devices become more intelligent, users have an increasing need for interaction with smart devices, such as gaming interaction between users and smart screens.

[0004] However, the current ways users interact with smart devices are simplistic and lack fun, and the user experience needs to be improved. Summary of the Invention

[0005] This application provides a method, electronic device, and system for human-computer interaction. Through this method, electronic device, and system, when the electronic device detects that a user clicks on a control in the current third-party application interface, the electronic device can issue corresponding feedback through its own system capabilities. In this way, the interaction feedback between the user and the electronic device does not depend on the third-party application, and can provide the user with timely and rich feedback, providing the user with an immersive interactive experience.

[0006] In a first aspect, a human-computer interaction method is provided, which is applied to a first device. The method includes: the first device displays a first interface of a first application, the first application being a third-party application; obtaining position information of one or more controls existing on the first interface through the system capabilities of the first device, the one or more controls including a first control; and determining, based on the position information of the first control, when a user performs an operation on the first control, the first device issues a first feedback.

[0007] Wherein, the first device obtains the position information of the control through the system capabilities of the first device, which means that the process of the first device obtaining the position information of the control is implemented by the system capabilities of the first device itself, without relying on third-party capabilities, such as the capabilities of third-party applications.

[0008] In this embodiment, when the electronic device detects that the user clicks on a control in the current third-party application interface, the electronic device can issue corresponding feedback through its own system capabilities. In this way, the interaction feedback between the user and the electronic device does not depend on the third-party application, and can provide the user with timely and rich feedback, providing the user with an immersive interactive experience.

[0009] In conjunction with the first aspect, in one possible implementation, the first feedback includes feedback associated with the first control emitted by the first device via display, sound, and / or vibration.

[0010] In this embodiment, when the electronic device detects that a user clicks on a control in the current third-party application interface, it can provide feedback through its own system capabilities via display, sound, and / or vibration. The feedback methods are rich, allowing users to perceive the interactive feedback more explicitly and providing them with an immersive interactive experience.

[0011] In conjunction with the first aspect, in one possible implementation, the method further includes: when a user operates on the first control based on the position information of the first control, the first device sends first instruction information to the second device, the first instruction information being used to instruct the second device to issue a second feedback, and a connection is established between the second device and the first device.

[0012] Optionally, the first indication information may carry first feedback information, and the first indication information may also be used to instruct the second device to issue second feedback based on the first feedback information. The first feedback information may include, for example, display information, sound effect information, and / or vibration information.

[0013] In this embodiment of the application, when a user interacts with an electronic device through a control device, the electronic device can also instruct the control device connected to it to send feedback synchronously, which can further provide the user with timely and rich feedback, enhance human-computer interaction in multiple dimensions, provide the user with an immersive interactive experience, and the timely and rich feedback can also help the user quickly optimize the interactive operation, so that the user can master the interaction skills more quickly.

[0014] In conjunction with the first aspect, in one possible implementation, the second feedback includes feedback emitted by the second device via display, sound, and / or vibration.

[0015] Optionally, the second feedback and the first feedback can have a linkage relationship, that is, the second feedback and the first feedback can exhibit a linkage effect.

[0016] In this embodiment, when the electronic device detects that a user clicks on a control in the current third-party application interface, it can instruct the control device connected to it to synchronously provide feedback through display, sound and / or vibration. This can further provide the user with timely and rich feedback, enhance human-computer interaction in multiple dimensions, provide the user with an immersive interactive experience, and the timely and rich feedback can also help the user quickly optimize the interactive operation, enabling the user to master the interactive skills more quickly.

[0017] In conjunction with the first aspect, in one possible implementation, obtaining the position information of one or more controls present on the first interface through the system capabilities of the first device includes: obtaining the position information of the one or more controls based on the image information of the first interface.

[0018] In this embodiment of the application, the electronic device can process the image information of the third-party application interface to obtain the position information of the controls existing in the third-party application interface, thereby enabling the electronic device to obtain the position information of the controls existing in the third-party application interface through the system capabilities of the electronic device.

[0019] In conjunction with the first aspect, in one possible implementation, the method further includes: the first device acquiring information sent by a second application for acquiring location information of a second control, the second application being a third-party application; and when it is determined based on the location information of the second control that a user is operating on the second control, the first device issuing a third feedback.

[0020] In this embodiment of the application, the electronic device can also obtain the position information of the controls existing in the interface of the third-party application from the third-party application, and then issue feedback through the system capabilities of the electronic device.

[0021] In conjunction with the first aspect, in one possible implementation, the first feedback includes the first control presenting a first special effect, the first device emitting a first prompt sound, and / or the first device emitting a first vibration.

[0022] In conjunction with the first aspect, in one possible implementation, the first control presents a first special effect including the first control being displayed in a floating state and / or the first control being displayed with color changing and flashing.

[0023] The first control's floating state display can refer to, for example, the first control being enlarged and displayed, or the first control being displayed floating on the first interface, such as being displayed floating above other controls.

[0024] In conjunction with the first aspect, in one possible implementation, the first prompt sound is associated with the function of the first control, and / or the first vibration is associated with the function of the first control.

[0025] In conjunction with the first aspect, in one possible implementation, the second feedback includes the second device displaying a second special effect, the second device emitting a second prompt sound, and / or the second device emitting a second vibration.

[0026] In conjunction with the first aspect, in one possible implementation, the second prompt sound is associated with the function of the first control, and / or the second vibration is associated with the function of the first control.

[0027] In conjunction with the first aspect, in one possible implementation, the user operates on the first control by either operating on the first control through a second device or operating on the first control through the first device.

[0028] In this embodiment of the application, the user can trigger the controls in the third-party application interface displayed on the electronic device through a control device connected to the electronic device, or the user can directly trigger the controls in the third-party application interface displayed on the electronic device through the electronic device.

[0029] In conjunction with the first aspect, in one possible implementation, the user operates on the first control through the second device, including: the user selects the first control through the second device, or the user performs a gesture to select the first control through the second device, or the user selects the first control while wearing the second device.

[0030] In conjunction with the first aspect, in one possible implementation, the user operates on the first control through the first device, including: the user selecting the first control on the first interface; the user selecting the first control through air gesture operation.

[0031] In conjunction with the first aspect, in one possible implementation, the method further includes: when determining, based on the position information of the first control, that a user is operating the first control, the first device sends first information to the first application, the first information being used by the first application to issue fourth feedback, the first information including the position information of the first control.

[0032] In this embodiment of the application, when the user clicks on a control in the current third-party application interface, the third-party application can also provide feedback synchronously, further enriching the feedback methods in the human-computer interaction process.

[0033] In conjunction with the first aspect, in one possible implementation, the first information further includes second instruction information, which is used to instruct the first operation information, and the first operation information is used by the first application to determine the manner of issuing the fourth feedback.

[0034] In this embodiment of the application, when a user clicks on a control in the current third-party application interface, the electronic device can inform the third-party application of the user's specific operation information on the control, so that the third-party application can determine how to send feedback.

[0035] In conjunction with the first aspect, in one possible implementation, the first operation information is virtual operation information.

[0036] In this embodiment of the application, when a user clicks on a control in the current third-party application interface, the electronic device informs the third-party application that the user's specific operation information on the control can be virtual operation information. In this way, the electronic device can control the way it sends feedback to the third-party application by changing the operation information.

[0037] In conjunction with the first aspect, in one possible implementation, the first device is any one of a smart screen, a smartphone, a tablet computer, a laptop computer, an in-vehicle screen, or a smart conference tablet.

[0038] In conjunction with the first aspect, in one possible implementation, the second device is any one of a remote control, a mouse, a gamepad, or a wearable device.

[0039] Secondly, a human-computer interaction method is provided, which is applied to a second device. The method includes: the second device receiving first instruction information sent by a first device, the first instruction information being used to instruct the second device to issue a second feedback, wherein the first instruction information is sent by the first device when it determines that a user is operating the first control based on the position information of the first control, and a connection is established between the second device and the first device; and the second device issuing the second feedback according to the first instruction information.

[0040] Optionally, the first indication information may carry first feedback information, and the first indication information may also be used to instruct the second device to issue second feedback based on the first feedback information. The first feedback information may include, for example, display information, sound effect information, and / or vibration information.

[0041] In this embodiment of the application, when a user interacts with an electronic device through a control device, the electronic device can also instruct the control device connected to it to send feedback synchronously, which can further provide the user with timely and rich feedback, enhance human-computer interaction in multiple dimensions, provide the user with an immersive interactive experience, and the timely and rich feedback can also help the user quickly optimize the interactive operation, so that the user can master the interaction skills more quickly.

[0042] In conjunction with the second aspect, in one possible implementation, the second feedback includes feedback emitted by the second device via display, sound, and / or vibration.

[0043] In this embodiment, when the electronic device detects that a user clicks on a control in the current third-party application interface, it can instruct the control device connected to it to synchronously provide feedback through display, sound and / or vibration. This can further provide the user with timely and rich feedback, enhance human-computer interaction in multiple dimensions, provide the user with an immersive interactive experience, and the timely and rich feedback can also help the user quickly optimize the interactive operation, enabling the user to master the interactive skills more quickly.

[0044] In conjunction with the second aspect, in one possible implementation, the second feedback includes the second device emitting a second prompt tone and / or the second device emitting a second vibration.

[0045] In conjunction with the second aspect, in one possible implementation, the second prompt sound is associated with the function of the first control, and / or the second vibration is associated with the function of the first control.

[0046] In conjunction with the second aspect, in one possible implementation, the second device is any one of a remote control, a mouse, a gamepad, or a wearable device.

[0047] In conjunction with the second aspect, in one possible implementation, the first device is any one of a smart screen, smartphone, tablet computer, laptop computer, in-vehicle screen, or smart conference tablet.

[0048] Thirdly, an electronic device is provided, comprising one or more processors and one or more memories, and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the method of the first aspect or any possible implementation thereof, or to perform the method of the second aspect or any possible implementation thereof.

[0049] Fourthly, a system is provided, comprising: a first electronic device for performing the method of the first aspect or any possible implementation thereof; and a second electronic device for performing the method of the second aspect or any possible implementation thereof.

[0050] Fifthly, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method of the first aspect or any possible implementation thereof, or implement the method of the second aspect or any possible implementation thereof.

[0051] In a sixth aspect, a chip is provided, wherein instructions are stored therein, which, when executed on a device, cause the chip to perform the method of the first aspect or any possible implementation thereof, or to perform the method of the second aspect or any possible implementation thereof.

[0052] In a seventh aspect, a computer program product is provided, which stores a computer program or instructions that, when executed, implement the method in the first aspect or any possible implementation thereof, or implement the method in the second aspect or any possible implementation thereof. Attached Figure Description

[0053] Figure 1A is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0054] Figure 1B is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0055] Figure 2 is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0056] Figure 3 is a schematic flowchart of a human-computer interaction method;

[0057] Figure 4 is a schematic flowchart of a human-computer interaction method provided in the embodiment of the application;

[0058] Figure 5 is a schematic flowchart of another human-computer interaction method provided in an embodiment of this application;

[0059] Figure 6 is a schematic flowchart of another human-computer interaction method provided in an embodiment of this application;

[0060] Figure 7 is a schematic diagram of an application scenario of human-computer interaction provided in an embodiment of this application;

[0061] Figure 8 is a schematic diagram of another human-computer interaction application scenario provided by an embodiment of this application;

[0062] Figure 9 is a schematic diagram of another human-computer interaction application scenario provided by an embodiment of this application;

[0063] Figure 10 is a schematic flowchart of a human-computer interaction system provided in an embodiment of this application. Detailed Implementation

[0064] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.

[0065] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0066] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0067] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0068] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0069] The methods provided in this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0070] Figure 1A is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application.

[0071] As shown in Figure 1A, the electronic device 100 includes a processor 201, a memory 202, and a communication interface 203, as well as a display screen 209, a camera 210, a universal serial bus (USB) interface 211, a high-definition multimedia interface (HDMI) 212, an audio module 213, a power supply module 214, and buttons 215.

[0072] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] The display screen 209 is used to display images, videos, etc. The display screen 209 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), such as an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 209, where N is a positive integer greater than 1.

[0074] Electronic device 100 implements display functions through a GPU, a display screen 209, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 209 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 201 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0075] In some examples, the electronic device 100 can perform corresponding touch functions through the display screen 209.

[0076] Camera 210 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 210, where N is a positive integer greater than 1.

[0077] Alternatively, the electronic device 100 may not be equipped with a camera 210.

[0078] USB interface 211 is an interface compliant with the USB standard specification, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 211 can be used for data transfer between electronic device 100 and peripheral devices. For example, electronic device 100 can establish a wired communication connection with electronic device 200 through USB interface 211. Alternatively, electronic device 100 can also establish a wireless communication connection with electronic device 200 through communication interface 203.

[0079] HDMI 212 can be used to transmit high-quality video signals and multi-channel audio signals, avoiding the quality loss of traditional analog signal transmission.

[0080] The audio module 213 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 213 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 213 may be located in the processor 201, or some functional modules of the audio module 213 may be located in the processor 201.

[0081] The power supply module 214 is used to supply power to the processor 201, memory 202, communication interface 203, display screen 209, camera 210, audio module 213 and other modules in the electronic device 100.

[0082] Buttons 215 include a power button, volume buttons, etc. Buttons 215 can be mechanical buttons or touch-sensitive buttons.

[0083] In some embodiments, the electronic device 100 may be, for example, a smart screen.

[0084] For example, Figure 1B shows a schematic diagram of the structure of an electronic device 200. The electronic device 200 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, GPUs, ISPs, controllers, memory, video codecs, DSPs, baseband processors, and / or NPUs. These different processing units may be independent devices or integrated into one or more processors.

[0087] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0088] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] USB interface 130 is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge electronic device 200, and can also be used for data transfer between electronic device 200 and peripheral devices. This interface can also be used to connect other electronic devices, such as AR devices.

[0090] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0091] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 200. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0092] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0093] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0094] Electronic device 200 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0095] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be LCD, OLED, AMOLED, FLED, MiniLED, MicroLED, Micro-OLED, QLED, etc. In some embodiments, electronic device 200 may include one or N displays screens 194, where N is a positive integer greater than 1.

[0096] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0097] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 200 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0098] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 200 can receive button input and generate key signal inputs related to user settings and function control of electronic device 200.

[0099] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0100] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0101] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with or separate from the electronic device 200.

[0102] It should be understood that the phone cards in the embodiments of this application include, but are not limited to, SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated circuit cards (UICC), etc.

[0103] In some embodiments, the electronic device 200 may be, for example, a central control screen, a television, a desktop computer, a laptop computer, a smart conference tablet, a vehicle screen, or a portable electronic device such as a mobile phone, a foldable screen, a tablet computer, a watch, a bracelet, or a smart home device such as a smart speaker, a smart screen, etc.

[0104] For example, Figure 2 shows a schematic diagram of the structure of electronic device 300. Electronic device 300 may include processor 310, communication module 320, internal memory 330, external memory interface 340, universal serial bus (USB) interface 350, power management module 360, etc.

[0105] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] Processor 310 may include one or more processing units, such as a modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0107] The controller can be the nerve center and command center of the electronic device 300. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0108] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. This memory can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0109] USB port 350 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 350 can be used to connect a charger to charge electronic device 300, and can also be used for data transfer between electronic device 300 and peripheral devices. This interface can also be used to connect other electronic devices.

[0110] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 300.

[0111] The power management module 360 ​​receives input from the battery and / or charging management module and supplies power to the processor 310, internal memory 330, external memory, and communication module 320. The power management module 360 ​​can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 360 ​​may also be located within the processor 310.

[0112] The communication function of electronic device 300 can be realized through communication module 320, modem processor and baseband processor, etc.

[0113] The electronic device 300 may further include a charging management module for receiving charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module may receive charging input from the wired charger via a USB interface 350. In some wireless charging embodiments, the charging management module may receive wireless charging input via the wireless charging coil of the electronic device 300. While charging the battery, the charging management module may also supply power to the electronic device via a power management module 360.

[0114] In other embodiments, the power management module 360 ​​and the charging management module may also be housed in the same device.

[0115] The external memory interface 340 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external memory interface 340 to perform data storage functions.

[0116] Internal memory 330 can be used to store computer executable program code, which includes instructions. Processor 310 executes various functional applications and data processing of electronic device 300 by running the instructions stored in internal memory 330. Internal memory 330 may include a program storage area and a data storage area.

[0117] In some embodiments, the electronic device 300 may be, for example, a control handle.

[0118] In some embodiments, the electronic device 300 may further include a sensor module, which may also be a remote control. The sensor module may include, for example, a gyroscope sensor, an accelerometer sensor, and / or a geomagnetic sensor.

[0119] It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

[0120] The technical solutions of this application embodiment can be applied to interactive electronic devices, for example. Exemplarily, they can be applied to televisions, desktop computers, laptops, in-vehicle screens, central control screens, and smart conference tablets; they can also be applied to portable electronic devices such as mobile phones, foldable screens, tablets, watches, bracelets, and 3D touch devices; they can also be applied to smart home devices such as smart speakers and smart screens; they can also be applied to control devices such as controllers, remote controls, and mice; and they can also be applied to electronic devices in future networks or in future evolved public land mobile networks (PLMNs). Specifically, this technical solution can be used in human-computer interaction scenarios when electronic devices are running third-party applications, such as a user interacting with a smart screen to play a third-party game using a game controller, a user interacting with a smart screen to control a third-party application interface using a directional remote control, or a user interacting with a smartphone or tablet to control a third-party application interface using gestures.

[0121] As home appliances become increasingly intelligent, users have a stronger need for interaction with these devices. For example, game interaction between users and smart screens has become a core element influencing the modern gaming experience.

[0122] Figure 3 shows a schematic flowchart of a human-computer interaction method 300. As shown in Figure 3, the method 300 includes:

[0123] S301: An electronic device displays a third-party application interface, which includes one or more controls.

[0124] S302: When an electronic device detects a user's click action, it sends the cursor position and the click action to a third-party application.

[0125] The user's click operation can be, for example, a click operation made by the user on the screen of an electronic device, or a click operation made by the user through a control device connected to the electronic device, such as a user pressing a button on a remote control or handle to trigger the click operation.

[0126] S303: The third-party application determines the control 1 clicked by the user based on the cursor position and the click operation, wherein the one or more controls mentioned above include control 1.

[0127] S304: Third-party applications send specific feedback.

[0128] In this method, when a user operates on a third-party application page of a smart device and clicks on controls in the interface, the interaction feedback between the user and the smart device relies entirely on the interaction mechanism of the third-party application. The smart device itself and the control device connected to it cannot provide feedback on the interaction. This makes the interaction feedback received by the user during the interaction with the smart device unclear. For example, in the scenario where the user is using the smart device to play an immersive game, the user cannot obtain rich and varied real-time feedback. Therefore, the current interaction method between the user and the smart device is monotonous and lacks fun, and the user's interactive experience needs to be improved.

[0129] In view of this, embodiments of this application provide a method, electronic device, and system for human-computer interaction. Through this method, electronic device, and system, the electronic device can identify specific controls in the current interface. When the user clicks on the specific control, the electronic device can issue corresponding feedback through its own system capabilities, without relying on third-party applications to issue feedback. For example, the specific control may present a floating color-changing effect, or the electronic device may emit vibrations or sound effects, providing the user with an immersive interactive experience.

[0130] Furthermore, when users interact with electronic devices through control devices, the electronic devices can also instruct the control devices connected to them to provide feedback through displays, sounds, and / or vibrations, providing users with timely and rich feedback, enhancing human-computer interaction in multiple dimensions, and providing users with an immersive interactive experience.

[0131] For example, Figure 4 shows a schematic flowchart of a human-computer interaction method 400 provided in an embodiment of this application. As shown in Figure 4, the method 400 includes:

[0132] The first screen of the first application is displayed on the first device's display screen.

[0133] The first application is installed on the first device. In some embodiments, the first application is a third-party application.

[0134] For example, third-party applications may include game applications, video applications, conferencing applications, etc.

[0135] S401: The first device obtains the position information of one or more controls existing on the first interface.

[0136] In some embodiments, the first device obtains the position information of one or more controls present on the first interface through the system capabilities of the first device.

[0137] Wherein, the first device obtains the position information of the control through the system capabilities of the first device, which means that the process of the first device obtaining the position information of the control is implemented by the system capabilities of the first device itself, without relying on third-party capabilities, such as the capabilities of third-party applications.

[0138] In one example, the first device obtains the position information of one or more controls present in the first interface based on the image information of the first interface.

[0139] The image information of the first interface can be acquired by an image acquisition device, such as by capturing the first interface with a camera; it can also be received by the first device from the first application through system capabilities; it can also be acquired by the first device itself according to the requirements of the first application through system capabilities; it can also be image information for displaying the first interface acquired by the first device through system capabilities, such as pixel information of the first interface; in addition, the image information of the first interface can be acquired by other means, such as by a third-party device and then sent to the first device, and this application does not limit this.

[0140] In some embodiments, the first device receives information sent by the first application for obtaining the position information of one or more controls existing on the first interface, and then the first device obtains the position information of one or more controls existing on the first interface based on the information for obtaining the position information of one or more controls existing on the first interface.

[0141] The information used to obtain the position information of one or more controls existing on the first interface may include, for example, the position information of each of the one or more controls, or it may include a marker map corresponding to the first interface, on which the position of each control is marked.

[0142] S402: When the user performs an operation on the first control based on the position information of the first control, the first device sends feedback 1.

[0143] Among them, one or more controls existing on the first interface include the first control.

[0144] One or more controls may be, for example, one or more functional controls that exist on the first interface.

[0145] For example, in a game application interface, the first control may be a function key such as a skill key, an attack key, or a directional key.

[0146] For example, in a short video browsing interface, the first control may be a control for switching videos, a control for collecting videos, a control for saving videos, a control for sharing videos, etc.

[0147] In some embodiments, the user's operation on the first control may include the user's operation on the first control through a second device or the user's operation on the first control through the first device, wherein the second device is a control device and a connection is established between the second device and the first device.

[0148] In some implementations, the user's operation on the first control via the second device may include, for example, the user selecting the first control via the second device, or the user performing a gesture to select the first control via the second device, or the user selecting the first control while wearing the second device.

[0149] For example, it could be an operation where the user controls the cursor displayed on the first interface to move to the first control through the second device; it could also be a method where the user controls the cursor displayed on the first interface to move to the first control through the second device and performs a click operation; it could also be an operation triggered by other controls executed through the second device, such as issuing a voice command to trigger the first control through the second device, etc., and this application does not limit it in this way.

[0150] The click operation may include, for example, a single click, a double click, a swipe, or a click operation in sequence, and this application does not limit it.

[0151] In some implementations, the user's operation on the first control via the first device may include, for example, the user selecting the first control on the first interface; or the user selecting the first control via air gestures; or other control-triggered operations executed by the first device, such as issuing a voice command to trigger the first control via the first device, etc. This application does not limit this.

[0152] In some embodiments, the user controls the cursor on the first device to move to the first control. For example, the user can control the cursor on the first device to move to the first control through a second device. The second device can be a gamepad, a mouse, a remote control, such as a pointing remote control, a three-dimensional touch device, a wearable device, or other devices that can control the movement of the cursor on the display screen. This application does not limit the scope of the invention.

[0153] In some embodiments, feedback 1 may include, for example, feedback associated with the first control emitted by the first device via display, sound, and / or vibration.

[0154] For example, feedback 1 includes a first control displaying a first effect, a first device emitting a first notification sound, and / or a first device emitting a first vibration.

[0155] In one example, the first device emits feedback associated with the first control through a display. This feedback may include the first control displaying a specific effect, such as a hover effect, a color-changing and / or blinking effect, a hover color-changing effect, a hover color-changing blinking effect, a control shape-changing effect, an effect that displays dynamic elements on the control, or other effects, which are not limited in this application.

[0156] In one example, the first device provides feedback associated with the first control via sound, such as prompting sound effects.

[0157] In one example, the first device emits feedback associated with the first control through vibration, such as the first device emitting vibrations with a specific amplitude and frequency.

[0158] In some embodiments, there is a correspondence between feedback 1 and the first control, that is, when the user triggers different controls, the corresponding interactive feedback is also different.

[0159] In one example, when a user triggers the first control, the interactive feedback may include a floating effect where the first control turns purple, and may also include a first sound effect or a vibration at a first frequency emitted by the first device; when a user triggers the second control, the interactive feedback may include a floating effect where the second control turns green, and may also include a second sound effect or a vibration at a second frequency emitted by the first device.

[0160] It is understood that the above-mentioned association of feedback 1 with the first control means that feedback 1 is at least one feedback made to the user's operation on the first control. Feedback 1 can be directly associated with the first control or indirectly associated with the first control.

[0161] In one example, feedback 1 is generated based on the function and / or shape characteristics of the first control. It can be assumed that feedback 1 is directly related to the first control. For example, if the first control is a control used to perform a favorite operation, feedback 1 is to dynamically display a favorite icon above the first control. Another example is if the first control is a control used to perform an attack operation, feedback 1 is to emit attack audio through the first device.

[0162] In one example, when feedback 1 is applied to the first control, it can also be considered that feedback 1 and the first control are directly related, for example, feedback 1 is to enlarge the display of the first control.

[0163] In one example, when feedback 1 is unrelated to the function and / or shape of the first control, and feedback 1 does not act on the first control, it can be considered that feedback 1 is indirectly related to the first control. For example, feedback 1 is to emit a general prompt sound or a general prompt vibration.

[0164] In this embodiment, the electronic device can obtain one or more controls on the current interface through its system capabilities. When it detects that a user clicks on any of the one or more controls, the electronic device can issue corresponding feedback through its own system capabilities, without relying on third-party applications. This provides the user with timely and rich feedback, including tactile feedback, visual feedback, and / or sound feedback. For example, the control may exhibit a floating color-changing effect, the electronic device may vibrate, or sound effects may be emitted. This can enhance human-computer interaction in multiple dimensions, providing users with an immersive interactive experience. The timely and rich feedback can also help users quickly optimize their interactive operations, enabling them to master interactive skills more quickly.

[0165] For example, Figure 5 shows a schematic flowchart of another human-computer interaction method 500 provided in an embodiment of this application. As shown in Figure 5, the method 500 includes:

[0166] The first screen of the first application is displayed on the first device's display screen.

[0167] The first application is installed on the first device. In some embodiments, the first application is a third-party application.

[0168] S501: The first device obtains the position information of one or more controls existing on the first interface.

[0169] In some embodiments, the first device obtains the position information of one or more controls present on the first interface through the system capabilities of the first device.

[0170] In one example, the first device obtains the position information of one or more controls present in the first interface based on the image information of the first interface.

[0171] In some embodiments, the first device receives information sent by the first application for obtaining the position information of one or more controls existing on the first interface, and then the first device obtains the position information of one or more controls existing on the first interface based on the information for obtaining the position information of one or more controls existing on the first interface.

[0172] The further explanation of this step is the same as the explanation of S401 in the embodiment shown in Figure 4, and will not be repeated here for the sake of brevity.

[0173] S502: When the user performs an operation on the first control based on the position information of the first control, the first device sends feedback 1, wherein one or more controls existing on the first interface include the first control.

[0174] The explanation of this step is the same as that of S402 in the embodiment shown in Figure 4, and will not be repeated here for the sake of brevity.

[0175] S503: When the user operates on the first control based on the position information of the first control, the first device sends a first instruction information to the second device. The first instruction information is used to instruct the second device to issue feedback 2, and a connection is established between the second device and the first device.

[0176] In some embodiments, the first indication information may carry first feedback information, and the first indication information may also be used to instruct the second device to issue feedback 2 based on the first feedback information. The first feedback information may include, for example, display information, sound effect information and / or vibration information.

[0177] The second device is a control device used to enable interaction between the user and the first device. The second device can be, for example, a handle, a remote control, a mouse, a wearable device, or a 3D touch device.

[0178] In some embodiments, feedback 2 may include feedback emitted by the second device via display, sound, and / or vibration.

[0179] For example, feedback 2 may include the second device displaying a second special effect, the second device emitting a second prompt sound, and / or the second device emitting a second vibration. The second special effect, the second prompt sound, and the second vibration may be associated with the function of the first control.

[0180] For example, feedback 2 may include feedback corresponding to feedback 1 emitted by the second device through display, sound and / or vibration.

[0181] In one example, the second device sends feedback corresponding to feedback 1 through a display. For example, when feedback 1 includes the first control presenting a specific effect, correspondingly, feedback 2 includes displaying the specific effect on the display screen of the second device. The specific effect may be, for example, a hover effect, a color-changing and / or blinking effect, a hover color-changing effect, a hover color-changing blinking effect, a control shape change effect, an effect of displaying dynamic elements on the control, or other effects, which are not limited in this application.

[0182] In one example, the second device emits feedback corresponding to feedback 1 via sound. For example, when feedback 1 includes a prompting sound effect emitted by the first device, correspondingly, feedback 2 also includes emitting that prompting sound effect.

[0183] In one example, the second device emits feedback corresponding to feedback 1 through vibration. For example, when feedback 1 includes the first device emitting vibration with a specific amplitude and frequency, correspondingly, feedback 2 also includes emitting the same vibration with a specific amplitude and frequency.

[0184] The explanation of the user's operation on the first control has been explained in detail in S402 of the embodiment shown in Figure 4, and will not be repeated here for the sake of brevity.

[0185] S504: Based on the first instruction information, the second device sends feedback 2.

[0186] The execution order of S502 and S503 is not limited. S503 can be executed first and S502 can be executed later; S502 can be executed first and S503 can be executed later; or S502 and S503 can be executed simultaneously.

[0187] From the user's perspective, S502 and S504 can be executed simultaneously, meaning the user simultaneously perceives feedback 1 from the first device and feedback 2 from the second device.

[0188] In this embodiment of the application, the electronic device can obtain one or more controls in the current interface through the device's system capabilities. When it detects that the user clicks on any of the one or more controls, the electronic device can issue corresponding feedback through its own system capabilities. It does not rely on third-party applications to issue feedback, and can provide users with timely and rich feedback, thus providing users with an immersive interactive experience.

[0189] Furthermore, electronic devices can instruct control devices to send feedback synchronously, providing users with timely and rich feedback, enhancing human-computer interaction in multiple dimensions, providing users with an immersive interactive experience, and timely and rich feedback can also help users quickly optimize interactive operations, enabling users to master interactive skills more quickly.

[0190] For example, Figure 6 shows a schematic flowchart of another human-computer interaction method 600 provided in an embodiment of this application. As shown in Figure 6, the method 600 includes:

[0191] S601: When the first device displays the first interface of the first application, the first device obtains the position information of one or more controls existing on the first interface.

[0192] The first application is installed on the first device. In some embodiments, the first application is a third-party application.

[0193] In some embodiments, the first device obtains the position information of one or more controls present on the first interface through the system capabilities of the first device.

[0194] In one example, the first device obtains the position information of one or more controls existing in the first interface based on the image information of the first interface. For example, the first device uses a specific algorithm to obtain the position information of one or more controls existing in the first interface based on the image information of the first interface. This specific algorithm may be embedded in the first device. When the user interacts with the first interface (for example, the user uses a game controller to control the game application interface on the first device to play a game), the first device begins to obtain the position information of one or more controls existing in the first interface through the specific algorithm.

[0195] In some embodiments, the first device receives information sent by the first application for obtaining the position information of one or more controls existing on the first interface, and then the first device obtains the position information of one or more controls existing on the first interface based on the information for obtaining the position information of one or more controls existing on the first interface.

[0196] In some embodiments, the first device may acquire the position information of one or more controls present in the first interface when the first interface is displayed.

[0197] S602: When the user performs an operation on the first control based on the position information of the first control, the first device sends feedback 1, wherein one or more controls existing on the first interface include the first control.

[0198] The explanation of this step is the same as that of S402 in the embodiment shown in Figure 4, and will not be repeated here for the sake of brevity.

[0199] S603: When it is determined that the user is operating the first control based on the position information of the first control, the first device sends the first information to the first application.

[0200] The first information is used by the first application to issue feedback 3, and the first information includes the position information of the first control.

[0201] In some embodiments, the first information further includes second indication information, which is used to indicate the first operation information, and the first operation information is used by the first application to determine the manner of issuing feedback 3.

[0202] In one example, feedback 1 is the first control floating and changing color, and feedback 3 is the first device emitting a vibration alert.

[0203] S604: Based on the first information, the first application sends feedback 3.

[0204] In some embodiments, the first information, including the first operation information, may be virtual operation information. The first operation information may be, for example, information used to instruct the user on a specific operation performed on the first control. Specific operations may include, for example, single-click, double-click, swipe, or sequential clicking. Different specific operations may correspond to different feedback methods; for example, a single-click operation corresponds to a control exhibiting a shaking effect, a double-click operation corresponds to a control exhibiting a floating zoom effect, and a swipe operation corresponds to a control exhibiting a blinking effect.

[0205] In one example, a single click corresponds to a control that exhibits a shaking effect, a double click corresponds to a control that exhibits a floating zoom effect, and a swipe corresponds to a control that exhibits a blinking effect. The user's action on the first control is a single click. The first information includes virtual operation information, which indicates that the user's action on the first control is a double click. Thus, after receiving the first information, the feedback 3 from the first application is the feedback that the control corresponding to the double click exhibits a floating zoom effect, rather than the feedback that the control corresponding to the single click exhibits a shaking effect.

[0206] In some embodiments, after receiving the first information, the first application determines the control triggered by the user as the first control based on the position information of the first control therein, and then issues feedback 3 to the first control.

[0207] For example, the first application has a pre-set correspondence table, which includes the correspondence between multiple controls in the application and multiple control position information. The first application can compare the control position information in the first information with the multiple control position information in the correspondence table to find the control that the user operates.

[0208] For example, feedback 3 may include feedback from the first application via display, sound, and / or vibration.

[0209] S605: When the user operates on the first control based on the position information of the first control, the first device sends a first instruction information to the second device. The first instruction information is used to instruct the second device to issue feedback 2, and a connection is established between the second device and the first device.

[0210] In some embodiments, the first indication information may carry first feedback information, and the first indication information may also be used to instruct the second device to issue feedback 2 based on the first feedback information. The first feedback information may include, for example, display information, sound effect information and / or vibration information.

[0211] In some embodiments, feedback 2 may include feedback emitted by the second device via display, sound, and / or vibration.

[0212] The further explanation of this step is the same as the explanation of S503 in the embodiment shown in Figure 5, and will not be repeated here for the sake of brevity.

[0213] S606: Based on the first instruction information, the second device sends feedback 2.

[0214] The execution order of S602, S603, and S605 is not limited; they can be executed simultaneously or sequentially.

[0215] From the user's perspective, S602, S604, and S606 can be executed simultaneously, meaning the user simultaneously perceives feedback 1 from the first device, feedback 2 from the second device, and feedback 3 from the first application.

[0216] In some embodiments, the feedback methods corresponding to feedback 1, feedback 2, and feedback 3 may have a linkage relationship, such as a linkage effect in display elements, vibration frequency, sound effects, etc.

[0217] In one example, feedback 1 corresponds to a floating control that changes color, feedback 2 corresponds to a vibration alert, and feedback 3 corresponds to a sound alert.

[0218] In some embodiments, a second application is installed on the first device. The second application is a third-party application. The first device also displays a second interface of the second application. The second interface includes a second control. The first device can obtain information sent by the second application to obtain the location information of the second control. Then, based on the location information of the second control, when it is determined that the user is operating the second control, the first device can issue feedback 4.

[0219] The first device can obtain information sent by the second application to obtain the location information of the second control from the data of the second application during the installation of the second application, or it can receive information sent by the second application to obtain the location information of the second control during the human-computer interaction process. For example, it can receive information sent by the second application to obtain the location information of the second control when the second interface is displayed during the human-computer interaction process.

[0220] The explanation for feedback 4 is the same as that for feedback 1, and will not be repeated here for the sake of brevity.

[0221] In this embodiment of the application, the electronic device can obtain one or more controls in the current interface through the device's system capabilities. When it detects that the user clicks on any of the one or more controls, the electronic device can issue corresponding feedback through its own system capabilities. It does not rely on third-party applications to issue feedback, and can provide users with timely and rich feedback, thus providing users with an immersive interactive experience.

[0222] Furthermore, electronic devices can instruct control devices to send feedback synchronously, providing users with timely and rich feedback, enhancing human-computer interaction in multiple dimensions, providing users with an immersive interactive experience, and timely and rich feedback can also help users quickly optimize interactive operations, enabling users to master interactive skills more quickly.

[0223] In addition, third-party applications can also provide feedback simultaneously, further enriching the feedback methods in the human-computer interaction process.

[0224] The following, with reference to Figures 7 to 9, will exemplarily introduce several application scenarios provided by the embodiments of this application.

[0225] For example, taking the interaction between a user and the game application interface displayed on the smart screen via a game controller as an example, Figure 7 shows a schematic diagram of an application scenario of human-computer interaction provided by an embodiment of this application.

[0226] As shown in Figure 7, the display screen of the smart screen 700 shows a game application interface. The current interface includes multiple controls. The user is controlling the cursor displayed on the smart screen 700 to play the game using the gamepad 710. When the user moves the cursor to the control 701 and performs an operation on the control 701 using the gamepad 710 (e.g., performing a click operation), the control 701 displays a special effect 1. This special effect 1 may include, for example, a color-changing effect, a floating effect, or a text reminder effect. In addition, the smart screen 700 can also emit a prompt sound 1, which may be a sound effect associated with the function of the control 701. Furthermore, the gamepad 710 can also emit a vibration prompt, for example, a vibration prompt with amplitude and other parameters associated with the function of the control 701.

[0227] In some embodiments, the smart screen 700 embeds a specific algorithm. When a user uses the gamepad 710 to control the cursor displayed on the smart screen 700's screen to play a game, the smart screen 700 can detect and identify controls present in the current game screen in real time according to the embedded algorithm, such as different skill controls, normal attack controls, directional controls, etc. When the smart screen 700 detects that the cursor position has moved to the identified control 1, and the smart screen 700 receives a click operation from the user, the control 1 displayed on the smart screen 700 will display a special effect 1, and / or the smart screen 700 will emit a prompt sound 1, and / or the gamepad 710 will emit a vibration prompt.

[0228] For example, taking the interaction between a user and a third-party application interface displayed on a smart screen via a directional remote control as an example, Figure 8 shows a schematic diagram of another human-computer interaction application scenario provided by an embodiment of this application.

[0229] As shown in Figure 8, the smart screen 800 displays a video application interface. The current interface includes multiple controls, such as a "Home" control, a "Radio" control, a "Video" control, and a "My" control. The user can control the cursor displayed on the smart screen 800 to move using a directional remote control 810. When the user moves the cursor to the "Home" control using the directional remote control 810 and performs an operation on the control 801 using the directional remote control 810 (e.g., clicking), the "Home" control displays a special effect 2. This special effect 2 can include, for example, a text magnification effect, a text magnification and blinking effect, a text color-changing and blinking effect, a control floating color-changing effect, or a heart-shaped blinking effect around the control. Furthermore, the smart screen 800 can also emit a notification sound 2, which can be a sound effect associated with the function of the control 801. In addition, the directional remote control 810 can also emit a vibration notification, such as a vibration notification with amplitude and other parameters associated with the function of the control 801.

[0230] In some embodiments, the smart screen 800 embeds a specific algorithm. When a user uses the pointing remote control 810 to move the cursor, the smart screen 800 can detect and identify controls present on the current screen in real time according to the embedded algorithm, such as the "My" control, "Recommended" control, "Like" control, and "Share" control. When the smart screen 800 detects that the cursor has moved to the identified control 2, and the smart screen 800 receives a click from the user, the control 2 displayed on the smart screen 800 will display a special effect 2 and / or emit a prompt sound 2, and / or the pointing remote control 810 will vibrate to indicate that the control is active.

[0231] In some embodiments, the directional remote control 810 can be replaced by a mouse.

[0232] In some embodiments, the directional remote control 810 can be replaced with a regular remote control.

[0233] In some embodiments, the directional remote control 810 can be replaced with a three-dimensional touch device.

[0234] The 3D touch device can be understood as a relatively advanced remote control. It may include a horizontally placed detection tablet. When the user performs gestures on the detection tablet, the detection tablet can recognize the user's gestures and their corresponding operation positions on the display screen. The user can control the cursor movement and perform click operations on the display screen by performing gestures on the detection tablet.

[0235] For example, taking the interaction between a user and a third-party application interface displayed on an electronic device through gestures as an example, Figure 9 shows a schematic diagram of another human-computer interaction application scenario provided by an embodiment of this application.

[0236] As shown in Figure 9, the screen of the electronic device 900 displays a video application interface. The current interface includes multiple controls, such as a "Home" control, a "Highlights" control, a "Special Zone" control, an "Education" control, and a "My" control. When the user interacts with the "Home" control (e.g., by clicking), the "Home" control displays a special effect 3. This special effect 3 may include, for example, a text magnification effect, a text font change effect, a text magnification and blinking effect, a text color-changing and blinking effect, a control floating color-changing effect, or a specific shape blinking effect around the control. Furthermore, the electronic device 900 can also emit a notification sound 3, which may be a sound effect associated with the function of the control 901. In addition, the electronic device 900 can also emit a vibration notification, such as a vibration notification with amplitude and other parameters associated with the function of the control 901.

[0237] Among them, electronic device 900 can be, for example, a smartphone, tablet computer or smart conference tablet, or other display device with touch function.

[0238] Among them, smart conference tablets can also be called smart collaboration tablets or smart conference office screens.

[0239] In some embodiments, the electronic device 900 embeds a specific algorithm. When a third-party application interface is displayed on the screen of the electronic device 900, the electronic device 900 can detect and identify controls present in the current interface in real time according to the embedded algorithm, such as controls for "My", "Highlights", "Games", and "Children". When the electronic device 900 detects that the user clicks on control 3 among the identified controls, the control 3 displayed on the electronic device 900 will display a special effect 3 and / or emit a prompt sound 3, and / or the electronic device 900 will vibrate to indicate that the control is active.

[0240] For example, Figure 10 shows a schematic flowchart of a human-computer interaction system 1000 provided in an embodiment of this application. As shown in Figure 10, the system 1000 includes a first device 1010 and a second device 1020, wherein the first device 1010 includes an algorithm module 1011, a display module 1012, and a sound effect module 1013, and the second device 1020 includes a vibration module 1021 and a sound effect module 1022. Specifically:

[0241] Algorithm module 1011 is used to obtain the position information of one or more controls existing on the first interface through the system capabilities of the first device when the first interface of the third-party application is displayed on the display screen of the first device 1010. The one or more controls existing on the first interface include control 1.

[0242] The algorithm module 1011 is also used to send instruction information to the display module 1012 and / or the sound effect module 1013 and / or the vibration module 1021 and / or the sound effect module 1022 of the second device 1020 when it is determined that the user is operating the control 1 based on the position information of the control 1. The instruction information is used to instruct the display module 1012 to display special effects for the control 1, and / or the instruction information is used to instruct the sound effect module 1013 to emit a sound effect associated with the control 1, and / or the instruction information is used to instruct the vibration module 1021 to emit a vibration prompt associated with the control 1, and / or the instruction information is used to instruct the sound effect module 1022 to emit a sound effect associated with the control 1.

[0243] The algorithm module 1011 can also be used to determine whether to enable the human-computer interaction method provided in this application for the first interface. When the algorithm module 1011 does not detect a control in the current interface (i.e., there is no functional space in the first interface), the algorithm module 1011 determines that the human-computer interaction method provided in this application will not be enabled for the first interface.

[0244] Algorithm module 1011 can also be used to determine whether to enable the human-computer interaction method provided in this application for the control 1 when the user operates on the control 1 based on the position information of the control 1. For example, when algorithm module 1011 determines, based on the interaction record between the first device and the second device, that the human-computer interaction method provided in this application has been enabled for the control 1 in a specific time period before the current time, algorithm module 1011 determines not to enable the human-computer interaction method provided in this application for the control 1.

[0245] The display module 1012 is used to present special effects on the control 1 after receiving the instruction information sent by the algorithm module 1011.

[0246] The sound effects module 1013 is used to emit sound effects associated with the control 1 after receiving the instruction information sent by the algorithm module 1011.

[0247] The vibration module 1021 is used to issue a vibration prompt associated with the control 1 after receiving the indication information sent by the algorithm module 1011.

[0248] The sound effects module 1022 is used to emit sound effects associated with the control 1 after receiving the instruction information sent by the algorithm module 1011.

[0249] In some embodiments, the display module 1012, sound effect module 1013, vibration module 1021, and sound effect module 1022 can also be used to send an interaction record to the algorithm module 1011 after the interaction ends. The interaction record can be used, for example, for the algorithm module 1011 to determine whether to enable the human-computer interaction method provided in this application for the control 1 when it determines that the user has operated on the control 1 based on the position information of the control 1; it can also be used for the algorithm module 1011 to adaptively update and optimize the algorithm.

[0250] In some embodiments, taking the example of a user interacting with a third-party application interface displayed on the first device 1010 by controlling the movement of a cursor displayed on the first device 1010, the algorithm module 1011 can also be used to send the cursor position and the user's click operation to the third-party application when the user operates on the control, so that the third-party application can confirm the triggered control and provide feedback.

[0251] In some embodiments, the first device 1010 may further include a vibration module for issuing a vibration alert when the user operates on the control 1 based on the position information of the control 1.

[0252] In some embodiments, the second device 1020 may further include a display module for displaying special effects associated with the control 1 when the user performs an operation on the control 1 based on the position information of the control 1.

[0253] Taking a scenario where a user controls a cursor displayed on a first device to play a game using a game controller as an example, in one implementation, a game application 1 is installed on the first device. This game application 1 is a third-party application. Based on the game application 1, an algorithm 1 adapted to the game application 1 is selected. Then, the algorithm 1 is adapted to the display module, sound effect module, and / or vibration module respectively. After the adaptation is completed, the algorithm module is set in the first device, and the algorithm corresponding to the algorithm module is algorithm 1.

[0254] In this way, by embedding a well-adapted algorithm module in the first device, the interaction methods of users in the process of playing third-party games on the first device can be enriched. Since users will receive obvious and rich tactile feedback after operating the controls, users can control the game interaction more and more accurately based on the feedback, thereby helping users to quickly optimize the interaction operation and improve the user's immersive gaming experience.

[0255] Taking a scenario where a user interacts with the third-party application display interface of the first device through a directional remote control as an example, in one implementation, a video application 1 is installed on the first device. This video application 1 is a third-party application. Based on the video application 1, an algorithm 2 adapted to the video application 1 is selected. Then, the algorithm 2 is adapted to the display module, the sound effect module, and / or the vibration module respectively. After the adaptation is completed, the algorithm module is set in the first device, and the algorithm corresponding to the algorithm module is algorithm 2.

[0256] In this way, by embedding a well-adapted algorithm module in the first device, the interaction methods of users when operating third-party video applications using control devices such as directional remote controls or 3D touch devices can be enriched. Since users will receive obvious and rich tactile feedback after operating the controls, it can accelerate the refinement of user habits. Users can control game interactions more and more accurately based on feedback, thereby helping users to quickly optimize interactive operations and enhance the user's immersive gaming experience.

[0257] Taking a scenario where a user interacts with the display interface of a third-party application on a first device via gestures as an example, in one implementation, a meeting application 1 is installed on the first device. This meeting application 1 is a third-party application. Based on the meeting application 1, an algorithm 3 adapted to the meeting application 1 is selected. Then, the algorithm 3 is adapted to the display module, sound effect module, and / or vibration module of the first device respectively. After the adaptation is completed, the algorithm module is set in the first device, and the algorithm corresponding to the algorithm module is algorithm 3.

[0258] In this way, by embedding a well-adapted algorithm module in the first device, the interaction methods of users when using third-party applications can be enriched. Since users will receive obvious and rich tactile feedback after operating the controls, it can accelerate the refinement of user habits. Users can control game interactions more and more accurately based on feedback, thereby helping users to quickly optimize interactive operations, improve interaction efficiency, and enhance the user's immersive gaming experience.

[0259] One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0260] When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

[0261] When the modules or units described herein are implemented using software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0262] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0263] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0264] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0265] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0266] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0267] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0268] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of human-machine interaction, characterized in that, The method is applied to a first device, and the method comprises: The first device displays a first interface of a first application, and the first application is a third-party application; The first device obtains position information of one or more controls existing on the first interface through a system capability of the first device, and the one or more controls comprise a first control; When it is determined that a user operates the first control based on the position information of the first control, the first device sends first feedback.

2. The method of claim 1, wherein, The first feedback comprises feedback associated with the first control sent by the first device in the form of display, sound and / or vibration.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: When it is determined that a user operates the first control based on the position information of the first control, the first device sends first indication information to a second device, and the first indication information is used to instruct the second device to send second feedback, and a connection is established between the second device and the first device.

4. The method of claim 3, wherein, The second feedback comprises feedback sent by the second device in the form of display, sound and / or vibration.

5. The method according to any one of claims 1 to 4, characterized in that, The first device obtains position information of one or more controls existing on the first interface through a system capability of the first device, and the one or more controls comprise a first control; According to the image information of the first interface, the position information of the one or more controls is obtained.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: The first device obtains information sent by a second application for obtaining position information of a second control, and the second application is a third-party application; When it is determined that a user operates the second control based on the position information of the second control, the first device sends third feedback.

7. The method according to any one of claims 1 to 6, characterized in that, The first feedback comprises that the first control presents a first special effect, the first device sends a first prompt sound and / or the first device sends a first vibration.

8. The method of claim 7, wherein, The first control presents the first special effect, which comprises that the first control is displayed in a floating state and / or the first control is displayed in a color-changing flashing manner.

9. The method according to claim 7 or 8, characterized in that, The first prompt sound is associated with the function of the first control, and / or the first vibration is associated with the function of the first control.

10. The method of claim 4, wherein, The second feedback comprises that the second device displays a second special effect, the second device sends a second prompt sound and / or the second device sends a second vibration.

11. The method of claim 10, wherein, The second prompt sound is associated with the function of the first control, and / or the second vibration is associated with the function of the first control.

12. The method according to any one of claims 1 to 11, characterized in that, The user operates the first control, which comprises: The user operates the first control through a second device or operates the first control through the first device.

13. The method of claim 12, wherein, The user operates the first control through the second device, which comprises: The user selects the first control through the second device, or the user performs a gesture of selecting the first control through the second device, or the user selects the first control by wearing the second device.

14. The method of claim 12, wherein, The user operates the first control through the first device, which comprises: The user selects the first control on the first interface; The user selects the first control through a gesture operation.

15. The method according to any one of claims 1 to 14, characterized in that, The method further comprises: When the user performs an operation on the first control based on the position information of the first control, the first device sends first information to the first application. The first information is used by the first application to issue fourth feedback, and the first information includes the position information of the first control.

16. The method of claim 15, wherein, The first information also includes second indication information, which is used to indicate the first operation information, and the first operation information is used by the first application to determine the method of issuing the fourth feedback.

17. The method of claim 16, wherein, The first operation information is virtual operation information.

18. The method of any one of claims 1 to 17, wherein, The first device can be any one of a smart screen, smartphone, tablet computer, laptop computer, in-vehicle screen, or smart conference tablet.

19. The method of claim 3 or 4, wherein, The second device is any one of a remote control, mouse, gamepad, or wearable device.

20. A method of human-machine interaction, characterized by The method is applied to a second device, and the method includes: The second device receives a first instruction message sent by the first device. The first instruction message is used to instruct the second device to issue a second feedback. The first instruction message is sent by the first device when it determines that the user is operating the first control based on the position information of the first control. A connection is established between the second device and the first device. Based on the first instruction information, the second device sends the second feedback.

21. The method of claim 20, wherein, The second feedback includes feedback emitted by the second device through display, sound, and / or vibration.

22. The method of claim 20 or 21, wherein, The second feedback includes the second device emitting a second prompt tone and / or the second device emitting a second vibration.

23. The method of claim 22, wherein, The second prompt sound is associated with the function of the first control, and / or the second vibration is associated with the function of the first control.

24. The method of any one of claims 20-23, wherein, The second device is any one of a remote control, mouse, gamepad, or wearable device.

25. The method of any one of claims 20-24, wherein, The first device can be any one of a smart screen, smartphone, tablet computer, laptop computer, in-vehicle screen, or smart conference tablet.

26. An electronic device, comprising: include: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as claimed in any one of claims 1 to 19, or to perform the method as claimed in any one of claims 20 to 25.

27. A system, comprising: include: A first electronic device is configured to perform the method as described in any one of claims 1 to 19; A second electronic device is used to perform the method as described in any one of claims 20 to 25.

28. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 19, or the method as described in any one of claims 20 to 25.

29. A chip, characterized by The chip stores instructions that, when executed, implement the method as described in any one of claims 1 to 19, or the method as described in any one of claims 20 to 25.

30. A computer program product, characterised in that, The computer program product stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 19, or the method as described in any one of claims 20 to 25.

Images